Symptoms vs Causes

At 03:00 the CPU on the database host climbs to 92%. The on-call engineer is paged. The engineer opens Grafana and sees a single spike that is now receding. Meanwhile, the user-visible check endpoint is still returning 200, and the checkout service is processing requests at the expected p95 of 180 milliseconds. The CPU spike was real; it was also benign. The on-call engineer lost thirty minutes of sleep for a non-incident.

Cause-based alerts (CPU above 80%) are noisy because they fire before user impact, sometimes by hours. Symptom-based alerts (latency above one second, error rate above 2%, availability probe failing) fire when users are already affected. Symptom alerts are rarer. Cause alerts are common. The platform should page on symptoms and ticket on causes.

What it is

A symptom is a user-visible failure: an HTTP 5xx, a checkout latency above the SLO, an availability probe failure, a missing metric in the synthetic check. The user, or the user’s proxy, sees the failure. A cause is an internal condition that may or may not produce a symptom: high CPU, low disk, elevated GC pause, slow log ingest. The cause is upstream of the symptom; the cause often precedes the symptom by minutes or hours, and many causes resolve before any symptom appears.

The dichotomy is the central design choice in any alerting stack. Page on symptoms. Investigate through causes. Ticket the causes that matter, delete the ones that do not.

Why a sysadmin cares

The dichotomy is what determines whether the on-call rotation is useful or burned out. A rotation that pages on causes spends the night chasing CPU spikes that resolve themselves. A rotation that pages on symptoms spends the night mitigating incidents the user actually felt. The first rotation mutates within a quarter. The second rotation learns the system.

How it works

The mental model is a tree. The symptom is the root the user sees. The causes are the branches and leaves. Telemetry that only describes the symptom (the HTTP 5xx rate) leaves the investigation at the top of the tree. Telemetry that reaches host and dependency metrics, structured logs, and traces reaches the bottom.

  Symptom (user-visible failure)
       |
  +----+-----+-----+
  |    |     |     |
 Slack  Probe  SLO   error
 reports fires breach  counter
  |    |     |     |
  +----+-----+-----+
       |
  Cause chain (internal conditions)
       |
  +----+-----+-----+-----+
  |    |     |     |     |
 CPU  disk  GC   pool   dependency
       I/O   pause size  latency

The investigation moves downward. The page fires on the symptom. The dashboards and traces walk downward through the causes. The on-call engineer acts on the cause that explains the symptom.

How to configure it

Two rules, same service, opposite routing. The first is a symptom alert on user-visible latency. The second is a cause alert on host CPU. The labels decide who gets woken up.

groups:
- name: checkout.rules
  interval: 30s
  rules:
  # SYMPTOM alert: latency above SLO. Pages.
  - alert: CheckoutLatencyAboveSLO
    expr: |
      histogram_quantile(0.95,
        sum by (le, region) (
          rate(http_request_duration_seconds_bucket{job="checkout"}[5m])
        )
      ) > 0.5
    for: 5m
    labels:
      severity: page
      team: payments
      service: checkout
      slo: latency
    annotations:
      summary: 'Checkout p95 latency above 500 ms for 5 minutes'
      description: |
        Region {{ $labels.region }} checkout p95 is
        {{ $value }}s. Users will feel this as a hang at
        submit. Investigate dependency latency or DB pool.
      runbook_url: 'https://runbooks.example.com/checkout/latency'
      dashboard_url: 'https://grafana.example.com/d/checkout'

  # CAUSE alert: host CPU high. Tickets.
  - alert: CheckoutHostCPUHigh
    expr: |
      100 - (avg by (instance) (
        rate(node_cpu_seconds_total{mode="idle",job="node"}[5m])
      ) * 100) > 85
    for: 30m
    labels:
      severity: ticket
      team: payments
      service: checkout
      slo: capacity
    annotations:
      summary: 'Checkout host CPU above 85% for 30 minutes'
      description: |
        Host {{ $labels.instance }} CPU is
        {{ $value | printf "%.0f" }}%. Investigate by opening
        the host dashboard; ticket if sustained.
      runbook_url: 'https://runbooks.example.com/host/cpu'

The latency rule has for: 5m and severity: page. It pages because users are already feeling the hang. The CPU rule has for: 30m and severity: ticket. It tickets because the host has headroom and the spike may resolve. The Alertmanager route tree routes severity: page to the on-call rotation and severity: ticket to the working-hours backlog.

The Alertmanager side:

# alertmanager.yml (excerpt)
route:
  receiver: default
  group_by: ['alertname', 'service']
  routes:
    - matchers:
        - severity = "page"
      receiver: pagerduty
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
    - matchers:
        - severity = "ticket"
      receiver: jira
      group_wait: 5m
      group_interval: 30m
      repeat_interval: 4h
receivers:
  - name: pagerduty
    pagerduty_configs:
      - service_key: '<redacted>'
  - name: jira
    webhook_configs:
      - url: 'https://jira.example.com/webhooks/observability'

How to validate it

Validate the rule syntax first:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/checkout.rules.yml

Unit-test both rules against fixture metrics. The latency rule should fire at the fixture timestamp where p95 crosses 500 ms; the CPU rule should not fire for the same fixture, demonstrating that the cause alert stays inert under healthy load.

# SEVERITY: READ-ONLY
promtool test rules test-checkout.yml

Verify the Alertmanager route resolves each label set to the correct receiver:

# SEVERITY: READ-ONLY
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname="CheckoutLatencyAboveSLO" \
  severity="page" team="payments" service="checkout"

Expected output ends with -> pagerduty. Repeat for severity: ticket; the route should end with -> jira.

Verify the routing in the live Alertmanager UI by inspecting the expanded configuration under Status -> Config. The two routes must be visible at the top of the tree.

How it can fail

Five failure modes recur when teams confuse symptoms and causes:

1. Cause alert on a symptom metric. The rule selects an internal metric (GC pause, thread count) and labels it severity: page. The metric moves constantly during deploys; pages fire on every deploy; engineers mute the notification.
2. Symptom alert on a cause metric. The rule selects HTTP 5xx rate and labels it severity: ticket. The real incident goes to Jira; nobody acts for nine hours.
3. Symptom alert with no for: clause. Latency spikes for thirty seconds during a cache warm; the rule fires; the on-call is paged for a non-incident. Add for: 5m and the spike stops paging.
4. Cause alert without a path to mitigation. CPU above 85% tickets, but no runbook tells the working-hours team what to do. The ticket accumulates; nobody closes it; the metric continues to rise; the next deploy saturates the host.
5. Symptom alert masked by a cause alert firing first. A cause alert on dependency latency pages, and Alertmanager inhibition suppresses the symptom alert that would have paged. The on-call fixes the dependency, but the user-facing checkout is still failing because the deploy was the cause. Inhibition is covered in a separate lesson; the symptom alert must survive the inhibition if it represents a distinct user impact.

How to troubleshoot it

When a page turns out to be on a cause metric (or a ticket turns out to be on a symptom metric), the order is:

1. Inspect the firing alert in the Alertmanager UI. Read the expr field of the rule. Decide whether the metric is user-visible (symptom) or internal (cause).
2. If the routing decision was wrong (page on cause or ticket on symptom), fix the label and reload. Do not silence.
3. If the metric itself is the wrong signal, replace the expr with one that measures the symptom (for cause alerts being paged) or the cause (for symptom alerts being ticketed).
4. Add a unit test that asserts the alert fires under the right condition and stays inert under the wrong one. The test is the guarantee that the next PR review will catch the regression.
5. Document the change in the team’s runbook repository. A change in routing is a change in on-call expectations; the runbook must reflect that.

Security implications

The symptom-vs-cause distinction has one security edge case: the metric you choose for the symptom may carry user-identifying information. An alert on http_requests_total{user_id="..."} is a privacy bug waiting to ship. The symptom metric should be aggregated: total error rate, p95 latency, request count, probe status. Never alert on a per-user metric; alert on the aggregate and use dashboards or traces to investigate the per-user impact.

Performance implications

Symptom alerts tend to evaluate cheap aggregates (sum, rate, histogram_quantile). Cause alerts tend to evaluate host metrics with higher cardinality (node_cpu_seconds_total across many cores, many hosts). The combination can dominate the rule evaluation budget if the cause alerts scan every core of every host. Two mitigations:

1. Pre-aggregate host metrics into recording rules (node:cpu_utilization:avg5m) and alert on the recording rule output. The alert evaluation becomes a trivial comparison on a small series set.
2. Use count by (region)(...) to bound the cardinality of the cause alert. The alert fires per region, not per host. The on-call engineer opens the region dashboard to find the specific host.

Production guidance

• The default for any new alert rule should be severity: ticket. Promote to severity: page only when the team has demonstrated that the alert represents user-visible failure that requires action within the SLO mitigation window.
• Dashboards are the bridge. The page surfaces the symptom; the dashboard provides the cause-chain walk. The dashboard_url annotation must be present on every page, even on tickets, so the working-hours team can investigate without re-typing the dashboard path.
• Symptom metrics should be SLI-derived. The lesson on SLIs / SLOs covers the relationship between user-visible metrics and the SLO error budget.
• Cause alerts should have a runbook and an owner. If the cause alert has no owner, the cause alert should be a dashboard panel, not an alert.

Verification

You should now be able to answer:

• What is the operational difference between a symptom metric and a cause metric, and where does each belong in the route tree?
• Why is a CPU alert on severity: page a design error, and why is a 5xx alert on severity: ticket a worse one?
• How does the for: clause interact with the symptom-vs-cause decision, and what is the right default?
• Why is the dashboard the bridge between symptom and cause, and what annotation makes that bridge clickable?

Quiz