Symptom definition and impact — the first two steps

The first two steps of the 11-step methodology are the slowest. They are the steps that are skipped under pressure. The discipline is to write the symptom down in one sentence and the impact in business terms, before any kubectl command.

Step 1: Define the symptom

The symptom is the observable fact. The symptom is not the cause; the symptom is the thing the user sees. The symptom is the input to step 2.

A symptom is well-formed when it:

• Names the object (Pod, Service, Node, PVC).
• Names the state (CrashLoopBackOff, 503, NotReady, Pending).
• Names the time (since when, how often).
• Does not name the cause.

A bad symptom:

“The new image is broken.”

A good symptom:

“The Pod billing-7d8f-abcde has restarted 12 times in the last 5 minutes, all with CrashLoopBackOff.”

The bad symptom is a hypothesis disguised as a fact. The good symptom is the observable evidence. The discipline is to write the symptom first, then move to the hypothesis.

flowchart TD
    A[Operator paged] --> B[Write symptom in one sentence]
    B --> C{Well-formed?}
    C -->|No| D[Rewrite]
    C -->|Yes| E[Move to step 2]

The first minute is the slowest minute. The operator who writes the symptom down is the operator who runs the methodology; the operator who skips the symptom is the operator who runs in panic.

Step 2: Determine the impact

The impact is the business consequence. The impact is not the technical detail; the impact is the customer-visible consequence. The impact is what drives the urgency.

The impact is well-formed when it:

• Names the users (the customers, the workload, the downstream).
• Names the degradation (latency, errors, complete outage).
• Names the scale (1% of requests, 100% of requests).

Examples:

• “5% of billing requests are failing; the customer-facing status is degraded.”
• “The billing-api Service is unreachable from checkout; checkout is failing.”
• “The cluster has 12 nodes; one is in DiskPressure and is rejecting new Pods.”

The impact is the input to step 3. The impact also decides whether the operator should stop and ship a mitigation, or continue investigating.

Why the first two steps are skipped

The first two steps are skipped under pressure because the operator is familiar with the cluster. The operator believes they know the symptom and the impact. The reality is:

• The operator’s familiarity is the assumption that the symptom is the same as yesterday.
• The operator’s familiarity is the bias that the impact is the same as the previous incident.

The discipline is to write the symptom and the impact down, every time, regardless of familiarity. The cost of writing down the obvious is one minute; the cost of skipping the obvious is a multi-hour outage.

Failure modes of skipping

The most common failure modes:

• Skipping the symptom. The operator runs kubectl get pods and begins to interpret the state. The interpretation is a hypothesis; the symptom is the observable evidence. The operator confuses the two.
• Skipping the impact. The operator focuses on the technical detail and forgets the business consequence. The operator fixes the wrong thing.
• Skipping both. The operator applies a known fix (“rollback the deployment”) without checking whether the fix applies to the current symptom. The fix is a guess.

The remediation is the discipline: write the symptom down, write the impact down, then start the investigation.

The role of the symptom and impact in the post-incident review

The symptom and the impact are the first two fields of the PIR. The PIR is the document that pays down the lesson; the symptom and the impact are the input to the timeline.

The symptom is the time-series of the cluster’s state. The impact is the time-series of the customer’s experience. The two are different narratives; the PIR is the moment when they converge.

Production discipline

The first two steps are the slowest. The discipline is the same scale-free: every incident runs the same workflow; the depth scales with the complexity. The symptom is the observable fact; the impact is the business consequence; the two are not the same.

• Write the symptom down in one sentence. The symptom is the observable fact; the cause is not the symptom.
• Write the impact down in business terms. The impact is the customer-visible consequence; the technical detail is not the impact.
• Use the PIR template. The symptom and the impact are the first two fields of the PIR.
• Never skip the first two steps. The cost of writing them down is one minute; the cost of skipping is a multi-hour outage.

Quiz