Hypothesis, test, restore, validate — the closing arc

The closing arc is: hypothesis, test, restore, validate. The arc converts a candidate cause into a closed incident. The discipline is to never skip a step. The arc is the only part of the methodology that changes the cluster’s state; the arc must be justified by the evidence.

Step 8: Hypothesis

The hypothesis is the proposed cause. The hypothesis is a single sentence that explains the symptom and predicts the outcome of the next test. The hypothesis is the moment when the operator stops gathering evidence and proposes a candidate.

The hypothesis is testable. A hypothesis that is not testable is a guess. The discipline is to write the hypothesis in the form “If X, then Y.”

Examples:

• “If the container is crashing because the new image is missing a library, then kubectl logs --previous will show ImportError: No module named 'requests'.”
• “If the Service is returning 503 because the EndpointSlice is empty, then kubectl get endpointslices will show no addresses.”
• “If the node is in DiskPressure because the kubelet’s garbage collection is failing, then du -sh /var/lib/kubelet will show unexpected growth.”

A hypothesis that is not testable is a guess. The operator who writes a guess is the operator who applies the wrong fix.

Step 9: Test

The test is the experiment that confirms or refutes the hypothesis. The test is the moment when the methodology produces evidence. The test is the only step that changes the cluster’s state.

flowchart TD
    A[Hypothesis] --> B[Test]
    B --> C{Confirmed?}
    C -->|Yes| D[Restore]
    C -->|No| E[Re-formulate hypothesis]
    E --> B
    D --> F[Validate]

The test is read-only when possible. The test is the command that confirms the hypothesis without changing the cluster’s state. The test is the input to step 10.

The most common failure modes of the test:

• Skipping the test. The operator applies the fix without testing. The fix is a guess; the cluster is at risk.
• Wrong test. The operator tests the wrong hypothesis. The test confirms nothing; the cluster is still failing.
• Test changes the state. The operator runs a write command (e.g., kubectl delete) as a test. The cluster’s state changes; the methodology is no longer read-only.

Step 10: Restore

The restore is the mitigation that brings the cluster back to a working state. The restore is not the fix; the restore is the rollback or the workaround. The fix is the follow-up that the post-incident review schedules.

The restore is mechanical. The restore is the documented procedure that the operator runs. The restore is the moment when the cluster’s state is restored.

Examples:

• kubectl rollout undo deployment/billing -n prod — rollback to the previous version.
• kubectl scale deployment/billing --replicas=0 -n prod — scale to zero to remove from the load balancer.
• kubectl cordon node-03 && kubectl drain node-03 --ignore-daemonsets --force — remove the node from the cluster.

The restore is the input to step 11.

Step 11: Validate

The validate is the evidence that the restore worked. The validate is the same mechanical checks that the post-change validation runs: KPIs, PDBs, synthetic traffic, workload Ready. The validate is the moment when the operator knows the incident is over.

kubectl get pods -n prod -l app=billing -o wide
kubectl get pdb -n prod -o yaml
kubectl logs -n prod -l app=billing --tail=20
for endpoint in billing-api; do
    curl -fsS -o /dev/null -w "%{http_code} %{time_total}\n" \
        https://${endpoint}.prod.example.com/healthz
done

The validate is the close-out. The validate is the evidence that the incident is over.

Closing the incident

The incident is closed when:

• The validation passes.
• The customer-facing status is restored.
• The on-call is paged off.
• The PIR is scheduled.

The PIR is the next step. The PIR is the artefact that pays down the lesson. The PIR is the input to the next change.

Failure modes of the closing arc

The most common failure modes:

• Skipping the hypothesis. The operator applies a fix without naming a cause. Nothing is proved, so a recurrence arrives with no explanation.
• Skipping the test. The operator assumes the hypothesis is correct and proceeds to the restore. The restore is unjustified.
• Skipping the validation. The operator closes the incident because the cluster ‘looks healthy.’ The validation is the evidence; the cluster is not closed.
• Skipping the PIR. The operator forgets the lesson. The next incident is the same incident.

Production discipline

The closing arc is the methodology’s deliverable. The discipline is to run the arc in order, never skip a step, and use the same commands every time. The arc’s output is the closed incident; the artefact is the PIR.

• Write the hypothesis in the form ‘If X, then Y.’ A hypothesis that is not testable is a guess.
• Run the test before the restore. The test is the evidence that the hypothesis is correct.
• Make the restore mechanical. The restore is the documented procedure that the operator runs.
• Validate with mechanical checks. The validation is the evidence that the incident is over.

Quiz