False Positive Investigation

The PagerDuty alert CheckoutHighErrorRate opens again. The on-call engineer has closed it without action three times in the last two weeks. The fourth time, the engineer decides to write down what happened, in what order, before doing anything else. The engineer opens a notepad titled “alert-name / today / first-time-investigating”.

Eight minutes later, the notepad reads:

1. Confirmed alert is firing (Prometheus /api/v1/alerts).
2. Ran rule expression as instant query. Value is 0.021,
   below the 0.05 threshold. So threshold is not the cause
   in this evaluation.
3. Read labels attached to the alert. instance=checkout-
   staging-eu-2 appears; also environment=staging. The
   rule targets environment=production. Layer 2.
4. Confirmed with: sum by (environment, instance)
   (rate(http_requests_total{...}[5m])) shows staging
   series at 0.18 5xx rate; production at 0.0042.
5. Hypothesis: rule is selecting staging traffic because
   the metric carries environment label but the rule
   never required it.
6. Test: write the rule with environment="production" as
   a positive matcher; promtool query instant returns only
   the production series at 0.0042.
7. Action: rule change to add the matcher; reload; alert
   clears within a minute. Close-out note: "Layer 2 fix;
   matcher added to exclude staging; threshold and for:
   unchanged."

That is a false-positive investigation. The shape is: confirm the alert, evaluate the expression against the data, identify the layer, form a hypothesis, test it, document the change.

What the investigation is

A false-positive investigation is the structured procedure for working out why an alert fired. It is not a debugging session in the alert’s underlying service; that is the incident investigation. The false-positive investigation establishes whether the alert was wrong (false positive) or the alert was right and the system is broken (true positive).

In a healthy team, every alert that pages a human produces an investigation note — even when the note reads “alert fired, condition confirmed, incident detected, escalation in progress, see runbook XYZ”. The note is the audit trail.

Why a sysadmin cares

The investigation is what protects the team’s trust in the alerting system. An investigation that ends without a documented note is a wasted trip to the dashboard; the next investigation starts with no accumulated knowledge. An investigation that ends with a tuned threshold and a written note is one cycle closer to an alerting system that catches real incidents at the right rate.

In a maturing team, the false-positive investigation is the highest-volume investigation the on-call engineer performs. A team that pages five times a day and writes five investigation notes per day will see the false-positive rate drop by half within a quarter. A team that pages five times a day and writes no notes will see the rate rise.

The procedure

The right investigation procedure has seven steps, in order. Each step produces evidence; the next step uses it.

+---------------------------+
| Step 1: confirm alert     |
+-------------+-------------+
              |
              v
+---------------------------+
| Step 2: evaluate          |
| rule expression as        |
| instant query             |
+-------------+-------------+
              |
              v
+---------------------------+
| Step 3: read labels on    |
| the firing alert          |
+-------------+-------------+
              |
              v
+---------------------------+
| Step 4: form hypothesis   |
| about which layer is wrong|
+-------------+-------------+
              |
              v
+---------------------------+
| Step 5: find evidence     |
| for the hypothesis        |
+-------------+-------------+
              |
              v
+---------------------------+
| Step 6: test the change   |
| against the data          |
+-------------+-------------+
              |
              v
+---------------------------+
| Step 7: ship and document |
+---------------------------+

Step 1. Confirm the alert is in the right state and carries the labels the team expects. The endpoint is /api/v1/alerts on the Prometheus server, or the Alertmanager UI. The alert object has labels, state, activeAt, and value. Read all four.

Step 2. Run the rule expression as an instant query. This step is the same as the diagnostic step in Lesson 01. The value of this query is the truth; everything else is interpretation.

Step 3. Read the labels attached to the alert. The alert’s labels are the labels the rule evaluated against. The label set is the most useful piece of evidence for identifying the layer. Layer 2 is overwhelmingly likely when the labels include values that no engineer recognises.

Step 4. Form a hypothesis about the layer that is wrong. Do this before collecting more data; the hypothesis narrows the data collection.

Step 5. Find evidence for the hypothesis. This is typically one more query: a sum by (label) for each label the rule is supposed to constrain, to enumerate the selected series set in full.

Step 6. Test the rule change against the data, before shipping. The shape is: rephrase the rule with the proposed change, run promtool query instant against the new rule expression, confirm the returned series set is the one the team intends.

Step 7. Ship the rule change. Document the investigation in the alert’s runbook log or in the team’s postmortem file. Note the layer that was wrong, the change that fixed it, and any follow-up actions.

The most common shape

The most common shape of a false-positive investigation is tune only the layer you have evidence for. The engineer suspects Layer 2 (Lesson 01, the dominant cause), confirms with a query, makes a Layer 2 change, documents it. The investigation takes ten minutes.

The second most common shape is tune the layer below the one that is wrong. The engineer suspects Layer 1 because the alert value looks low, raises the threshold, and the same condition fires next week at a higher number. The cost is the next investigation; the next engineer still finds the Layer-2 cause.

The third most common shape is close without an investigation. The engineer silences the alert, closes the page, returns to work. The audit trail is empty. The next time the alert fires, the next engineer starts from zero.

The shape to avoid is the second one. The shape to reject outright is the third.

Under the hood

The alert object in Alertmanager 0.28.x carries the state the alerting engine produced: firing or resolved, plus silencedBy, inhibitedBy, and a receivers array naming the routes that delivered the notification. Reading the status block is part of the investigation:

{
  "labels": {
    "alertname": "CheckoutHighErrorRate",
    "severity": "page",
    "service": "checkout",
    "environment": "production",
    "region": "us-east-1"
  },
  "annotations": {
    "summary": "Checkout 5xx rate above 5% in us-east-1",
    "runbook": "https://runbooks.example.com/checkout/high-error-rate"
  },
  "startsAt": "2026-08-13T03:14:22.014Z",
  "endsAt": "0001-01-01T00:00:00Z",
  "generatorURL": "http://prometheus:9090/graph?...",
  "status": {
    "state": "active",
    "silencedBy": ["silence-9f3a"],
    "inhibitedBy": [],
    "mutedBy": []
  },
  "receivers": ["pagerduty-prod", "slack-prod-alerts"]
}

Three fields change the investigation:

• silencesBy lists active silences. If silencedBy is non-empty, the alert is silenced; the investigation is why the silence was in place at firing time.
• inhibitedBy lists active inhibitions. If inhibitedBy is non-empty, the alert is being suppressed by another alert. The investigation is the parent alert.
• receivers lists the routes that did or would have delivered the notification. The investigation maps the alert name to the actual delivery.

Most false positives are caught at labels, not at status, but the status fields are necessary evidence when silence or inhibition is suspected.

How to configure it

The investigation produces no Prometheus configuration. The team’s investigation discipline is operational, not configuration. The closest configuration is the documentation convention: every alert name maps to a runbook file in the team’s repository, the runbook has a section called Investigation log, and every alert closure writes one entry to the section.

A typical runbook entry, in the team’s repository:

- alert: CheckoutHighErrorRate
  runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The runbook page:

# Runbook: CheckoutHighErrorRate

## Symptom
Paging alert when checkout 5xx rate sustained above 5%.

## Investigation log

### 2026-08-13 03:14
Layer 2. Rule selected the staging service because the
environment matcher was missing. Added environment matcher;
rel. Threshold and for: unchanged.

### 2026-07-30 11:42
Layer 1. Threshold was set at 0.05 against a service whose
normal band is 0.04. Lifted to 0.06 after the team reviewed
the 30-day distribution. For: unchanged.

### 2026-07-12 19:01
True positive. Real outage. Refs incident #1342.

Each entry names the layer. Future engineers read the runbook and see that the alert has been investigated before, the last fix was Layer 2 in August, and the next investigation that finds a Layer 1 cause is a meaningful regression.

How to validate it

Validate by completing the seven steps, in order, and closing with a one-paragraph note.

curl -s 'http://prometheus:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="CheckoutHighErrorRate")'

Run the rule expression:

promtool query instant \
  http://prometheus:9090/api/v1/query \
  '<rule expression here>'

Test the proposed change before shipping:

promtool query instant \
  http://prometheus:9090/api/v1/query \
  '<new rule expression here>'

Run the rule check:

promtool check rules /etc/prometheus/rules/checkout.yml

SUCCESS: rule files validated; 12 rules found, 0 errors

The investigation is complete when the one-paragraph note is written, the rule change is in git, and the investigation timestamp matches a moment when the alert was firing.

How it can fail

Six specific shapes, each with the symptom that distinguishes it from other investigation failures:

1. Investigate at the wrong layer. The engineer suspects Layer 1 because the alert value is below threshold during the investigation, raises the threshold, and the alert fires next month at a higher number on a Layer 2 cause. Symptom: the close-out note says “raised threshold” and the next false positive is in the same rule with the same labels.

2. Skip step 3 (read labels). The engineer reads the rule, sees a high count of up==0 examples in the history, assumes a target failure, and investigates the targets. The labels attached to the firing alert would have shown a region value that no engineer recognised. Symptom: the engineer spent 30 minutes looking at the wrong subsystem.

3. Skip step 6 (test before shipping). The engineer changes the rule on production and reloads Prometheus without a query check. The new rule produces empty vectors and silences the alert entirely. Symptom: the rule stops firing and the team does not know whether the alert is fixed or broken.

4. Close without a note. The engineer silences the alert and returns to work. The audit trail is empty. Symptom: the next engineer has no context for the silence; the runbook investigation log is empty.

5. Mix up the alert’s labels with the rule’s matchers. The engineer reads the rule’s matchers, concludes that the rule selects production, and ignores the alert’s labels which show environment=staging. Symptom: the engineer’s hypothesis was about the rule but the alert was firing on different labels.

6. Investigate without an active alert. The engineer reads the rule file and predicts the false positive without checking whether the alert is currently firing. Symptom: the investigation produces a conclusion about a state that no longer exists.

How to troubleshoot it

1. Start the investigation only when an alert is firing. The investigation is a reaction to a real alert, not a theoretical exercise.
2. Read the alert object first. Labels, status, silencedBy, inhibitedBy, receivers. The five fields together produce the alert’s fingerprint.
3. Run the rule expression as an instant query. Read the value against the threshold.
4. If the value is below threshold, list the labels. If the labels include a value that no engineer recognises, suspect Layer 2. Otherwise suspect Layer 1.
5. If the value is above threshold on the right series set, look at the timing. Layer 3 may be wrong.
6. Apply the rule change in a test expression first, then in the rule file, then reload.

Security implications

The investigation note in the runbook is visible to anyone with read access to the team’s repository. The note should not include labels or annotations that the team considers sensitive. If an alert name carries customer or infrastructure identifiers, the runbook name should be sanitised to a stable alias.

The Alertmanager UI exposes the alert object including the full label set. If the label set includes secrets (HMAC keys, customer IDs), the alert template is leaking them through the investigation. The lesson’s analogue in the alert-template discipline is to keep labels bounded.

Performance implications

There is no Prometheus performance cost to a well-conducted investigation. A poorly-conducted investigation can flood the rule history log if the engineer “test” the change in production without promtool query instant first.

A team with a high false-positive rate should also check the rule evaluation cost of the mis-tuned rule. A mis-tuned rule often selects more series than the corrected rule, and the cost is paid every evaluation interval.

Production guidance

• Run the seven steps in order. Do not collect evidence before the hypothesis.
• Test the rule change against promtool query instant before shipping.
• Document every investigation. Even a “no action, alert confirmed true positive” entry is useful.
• Re-use the close-out paragraph in the next reviewer’s onboarding.

Verification

You should now be able to answer:

• What are the seven steps of a false-positive investigation, in order?
• Why is the rule expression as an instant query the authoritative evidence, not the alert’s value display in Grafana?
• What does the status.silencedBy field of an Alertmanager alert object tell the investigator?
• Why is “skip step 6 (test before shipping)” a worse mistake than “skip step 7 (document)”?

Quiz