Symptoms, Evidence, and Causality

A symptom is what an operator or user observes. A cause is the upstream event the symptom depends on. Every investigation travels from the symptom to the cause. The signal tells you the symptom is present; the evidence tells you which of several plausible causes is the actual one.

This lesson is the longest part of the course’s foundation. It is not about Prometheus, Grafana, Loki, or Tempo. It is about how an investigator uses the platform those tools provide.

The investigation loop

Symptom
   |
Quantify impact
   |
Form a hypothesis
   |
Find evidence
   |
Test the hypothesis
   |
Validate or reject
   |
Locate the cause
   |
Restore service
   |
Document

The shape is intentional. It is the same shape regardless of signal. The investigator who skips “form a hypothesis” before “find evidence” is collecting evidence without a question, and will eventually learn that this approach is slower and less reliable than the question-first approach.

Symptoms

A symptom is a fact. It is what an engineer or user observed:

• “Checkout returned 500 for 30% of attempts between 02:50 and 03:10.”
• “Disk usage on db-02 climbed from 75% to 92% in 10 minutes.”
• “p99 latency of /api/orders spiked from 200 ms to 4 s.”

The symptom is observable. It is also quantitative or specific enough to drive investigation. “The system feels slow” is a weak symptom. “p99 latency above 4 s on /api/orders” is a strong symptom.

A symptom is not a cause. It is a destination at the top of a tree; the cause is at the bottom. The investigation is the walk down.

Quantifying impact

Before forming a hypothesis, the investigator needs to know what the impact actually is. Common metrics:

• User-visible. Error rate, transaction failure, business metric (orders / minute, payments / minute).
• Service-visible. Latency p99, queue depth, dependency errors.
• Host-visible. CPU, memory, disk, network.

The impact is the gateway between “is this real?” and “what do we do?” A user-visible metric that is far out of band is a page. A service-visible metric that is bounded is a ticket. The question is answered by the metric, not by gut feel.

Forming a hypothesis

The investigator produces 1-3 candidate causes. The candidates are usually drawn from operational history:

• Recent change (“the new connection pool config went out at 02:45, payment failures started at 02:50”).
• Known fragile component (“the S3 backing store has been a source of latency spikes”).
• Symptomatic behaviour that matches a previous incident (“this looks like the cache-stampede we saw last month”).

A hypothesis without evidence is a guess. A guess without a plausible cause is a hypothesis. The investigator picks the most likely hypothesis, knowing they will reject it if the evidence does not support it.

Finding evidence

The investigator pulls the relevant telemetry for the time window of the symptom:

• Service metrics. Application-level: request rate, latency, errors. Dependency metrics: latency, errors per dependency.
• Host metrics. CPU, memory, disk, network. Containers and hosts separately.
• Logs. Structured logs filtered by severity, route, time. Logs from the suspected dependency. Logs filtered by trace ID if a trace is available.
• Traces. A trace of one of the failing requests. A trace of a slow request from the same window.
• Change log. Recent deployments, configuration changes, database migrations.

The evidence is gathered in service of the hypothesis. A question-first investigator knows what to pull; a question-last investigator pulls everything and gets lost.

Testing the hypothesis

For each candidate, the investigator asks:

• Does the evidence support this candidate?
• Does the evidence contradict another candidate?
• Is there a piece of evidence I cannot yet produce?

A common mistake is to test only the favoured hypothesis. The investigator should attempt to disprove each candidate first, because that is when additional evidence is most informative.

The hypothesis is supported if at least three pieces of independent evidence correlate with the symptom. It is rejected if any of them contradict it. A single correlation is suggestive but insufficient.

Validating or rejecting

If the hypothesis is supported: act. Mitigate the symptom (revert the change, drain a queue, restart a service). Validate that the symptom metric returned to its baseline. Document the runbook you used; cross-link to it from the alert.

If the hypothesis is rejected: roll back the investigation trees you built, document what was rejected and why, and form the next hypothesis. Do not be tempted to “fix things you saw” during investigation. Unrelated state changes are themselves a failure mode.

Common red herrings

Four patterns appear repeatedly in failed investigations:

1. Coincidence. A deployment happened in the same window as the symptom but was unrelated. The investigator reverts a good change and the symptom persists.
2. Cascade. Multiple failures layered: the user-impact metric was elevated by a dependency that was, in turn, slow because of network congestion. The investigator fixes the dependency metric (correctly) but the network congestion is the real cause.
3. Symptoms of a symptom. A spike in error rate produces more error logs, which produces a log-rate alert, which produces a memory pressure alert on the log shipper. The investigator chases the log-rate alert and misses that the original error rate is what matters.
4. Recency. The investigator privileges the most recent change because it is most salient in memory. The actual cause is something older and less salient.

Red herrings are not avoided by being more clever; they are avoided by following the loop. Form hypothesis, find evidence, test, validate. The loop catches its own mistakes when used as designed.

Restoring service

The investigator’s first obligation is to mitigate the symptom quickly. Reverting a known change, draining a queue, restarting a service, or failover to a backup system are all valid mitigations. The investigation continues alongside the mitigation.

A common mistake is to delay mitigation until the cause is fully understood. By the time the cause is located, the user has been impacted for hours. Mitigate first, understand afterward. Document both.

Documenting

A short incident document captures:

• Symptom, impact, time window.
• Hypotheses considered.
• Evidence gathered (with links to panels / queries).
• Cause located.
• Mitigation applied.
• Verification that the symptom ended.
• Hypotheses rejected and why (so the next investigator does not repeat them).
• Follow-up actions (preventive controls, alert tuning).

The document is the evidence of evidence. It is also the ground truth for post-incident review. Without it, the next similar incident will be investigated from zero.

Production guidance

• Form the hypothesis in writing. A Slack thread, a comment on the alert, an issue description. The act of writing forces precision.
• Use three pieces of independent evidence to support a hypothesis.
• Mitigate the symptom before locating the cause. Document both.
• Save the timeline. The next investigator will read it.
• After resolution, capture the runbook and reference it from the alert. The course returns to this in Part CXIII.

Verification

You should be able to answer:

• What is the difference between a symptom and a cause?
• Why does forming the hypothesis before finding the evidence improve the speed and accuracy of investigation?
• How do you avoid red herrings?
• Why does mitigation precede full root-cause analysis?

Quiz