False Positive Anatomy

At 03:14 a PagerDuty alert opens: HighErrorRate on checkout-svc. The on-call engineer walks to a laptop, opens Grafana, and asks the right first question. Is the error rate actually high right now, or did the rule fire on noise? Three minutes later the answer is clear. The error rate is 0.4%, well below the 5% threshold, no traffic anomaly, no deploy. The rule selected a series whose instance label points at an old autoscaling group that no longer exists. The alert was real. The condition was not. The page was wasted and the trust in the rule is now slightly lower than it was before.

That is a false positive in production terms. It is not a bug in Prometheus. It is a contract violation between the rule and the reality it describes, and the on-call engineer is now paying for it.

What a false positive is

A false-positive alert is an alert in the firing state whose underlying condition is not actually broken when a human investigates. Three things distinguish it from a true positive:

1. The metric underlying the expression is within its normal operating range at the moment of investigation.
2. No correlated symptom (user report, deploy, dependency failure, change log) lines up with the firing time.
3. The rule, on inspection, fires on a label set that the operator did not intend to alert on, or against a threshold that does not match the system’s normal band.

The first two conditions are evidence. The third is the cause. A firing alert that satisfies conditions one and two but turns out to fire on a real misbehaviour that nobody has yet seen is not a false positive. The definition is “alert fired, condition checked, no incident found” — and the check is human.

A false positive is not the same as a noisy alert. A noisy alert fires often and the on-call engineer has stopped reading them. A noisy alert can be a true positive that the team has learned to dismiss. A false positive is specifically one where investigation returns no incident.

Why a sysadmin cares

Three costs accumulate from a rule with a high false-positive rate:

• Cognitive cost. Every page shifts attention and interrupts the on-call engineer. A team that pages ten times a night and finds nothing nine times stops looking on the eleventh.
• Trust cost. Real alerts get dismissed. A genuine outage begins with “this is the same rule that fired three times last week for no reason” and ends with an hour of delay.
• Diagnostic cost. Investigation time is the long tail. The one time in ten it actually is an incident, the on-call engineer treats it as the ninth false positive and walks the wrong path for twenty minutes.

In a healthy platform, fewer than one in twenty pages results in human-only investigation with no action taken. Higher than that and the team must tune. Lesson 06 walks the discipline that drives that ratio below five per cent.

The diagnostic order

There are three layers to a Prometheus alerting rule, and three corresponding failure shapes. The order to check matters because each layer has a faster validation step than the next.

        +-----------------------+
        |  Alert fires          |
        +----------+------------+
                   |
                   v
        +-----------------------+
        | Layer 1: THRESHOLD    |
        | Is the value actually |
        | above the threshold   |
        | right now?            |
        +----------+------------+
                   |  no -> look upstream
                   v
        +-----------------------+
        | Layer 2: AGGREGATION  |
        | Does the matcher /    |
        | grouping select the   |
        | intended series set?  |
        +----------+------------+
                   |  no -> look upstream
                   v
        +-----------------------+
        | Layer 3: TIME WINDOW  |
        | Did the sustained-    |
        | breach logic in `for:`|
        | match the expected    |
        | duration?             |
        +----------+------------+
                   |
                   v
        Find the cause; tune the layer where it sits

Layer 1 is the cheapest. Query the metric in Grafana with the exact expression from the rule body and read the value yourself. If the value is below the threshold and no recording rule is masking the truth, the threshold or the recording rule is wrong. If the value is above the threshold but the alert still does not represent a problem, the threshold is mis-set against the system’s normal band.

Layer 2 is the aggregation shape. The rule fires on a label combination that no longer represents the system (an old instance, a deprecated region, a job that has been merged into another). When the matching labels attached to the alert include a hostname no engineer recognises, aggregation is the layer to fix.

Layer 3 is the for: clause. The expression returns a value above threshold but only across a brief sample window, and the for: value is shorter than the natural duration of the dip. Lesson 03 walks this layer in detail.

The most common cause

In practice, in roughly half of the false positives the team reviews, the cause sits at Layer 2 (aggregation). Common shapes:

• The rule groups by job and the job label was renamed when the team moved to a new exporter configuration.
• The rule groups by instance and the deployment replaced per-host instances with per-pod instances under Kubernetes, so the metric series set multiplied and avg(...) no longer matches the team’s mental model.
• The rule uses sum by (service) (rate(...)) but the underlying metric has both service and service_name labels that do not align, so the sum includes series that should have been excluded by another label.
• The rule has no ignore clause, so a deliberate test fixture or a synthetic probe is treated as production traffic.

Each of these is a Layer-2 fix, not a Layer-1 or Layer-3 fix. Raising the threshold or extending for: masks the symptom without addressing it. The next similar condition will produce the same false positive at a slightly higher number or after a slightly longer wait.

Under the hood

An alerting rule in Prometheus is a stored expression that is evaluated on the rule evaluation interval (default 1m) for the lifetime of the process. For each combination of metric and labels that the expression selects, Prometheus tracks a state per series:

[inactive] --(expression returns a series for the labels)
   |
   v
[pending]  --(has remained in the result for the `for:` duration)
   |
   v
[firing]   --(sent to Alertmanager)

for: is not a smoothing window on the metric value. It is a minimum dwell time in the pending state before the alert is allowed to advance to firing. A rule with for: 5m against a metric that crosses threshold at 03:00 and returns to normal at 03:04 will go pending and will not advance — the pending entry for that label combination is cleared at the next evaluation when the expression stops returning a series for those labels.

When the alert reaches firing, Prometheus writes it to the notification log (the WAL) and Alertmanager picks it up. From that point the responsibility moves to Alertmanager: grouping, silencing, inhibition, routing.

The three layers align with three parts of this flow:

• Threshold lives in the expression body (> 0.05).
• Aggregation lives in the expression itself: which labels are selected, which are summarised, which are ignored.
• Time window lives in for: and the rule evaluation interval.

Layer 2 is most often the cause because labels drift more often than thresholds. An accidental label drift happens in a single configuration change. Threshold changes are usually discussed and reviewed. for: is usually deliberate.

How to configure it

For a rule that misfires on the wrong series, the configuration fix is in the expr. The pattern is to add a label matcher that excludes the misbehaving series from the input the rule selects. Real annotated rule:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        # Layer 2 fix: drop the synthetic probe job from
        # the series before the rule selects from them.
        expr: |
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"5..",
                environment="production",
                synthetic!="true",
              }[5m]
            )
          )
          /
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"2..|3..|4..|5..",
                environment="production",
                synthetic!="true",
              }[5m]
            )
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout error rate above 5% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The line synthetic!="true" is the Layer-2 fix. The other matchers (job=, environment=) define the population the threshold is intended to apply to; if any of those drift, the rule fires on a different population than the team expects.

How to validate it

Three checks, in order, before a reload:

promtool check rules /etc/prometheus/rules/checkout.yml

Sample output, after a successful check:

SUCCESS: rule files validated; 14 rules found, 0 errors

Then load the rule and query the expression directly. The expression from the rule body returns the value Prometheus will evaluate against the threshold and for::

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'sum by (service, region) (
     rate(
       http_requests_total{
         job="checkout-svc",
         code=~"5..",
         environment="production",
         synthetic!="true",
       }[5m]
     )
   ) /
    sum by (service, region) (
     rate(
       http_requests_total{
         job="checkout-svc",
         code=~"2..|3..|4..|5..",
         environment="production",
         synthetic!="true",
       }[5m]
     )
   )'

The result lists the series that will be evaluated against the threshold. If a series the team intends to catch is missing, the expression is wrong. If a series the team does not intend to catch is present, the expression is also wrong — that is what just caused the false positive.

Finally, check the alert state of the rule in production:

curl -s http://prometheus:9090/api/v1/alerts \
  | jq '.data.alerts[] | select(.labels.alertname=="CheckoutHighErrorRate")'

If the rule is firing on region=us-east-2 synthetic=true and not on the production series, the expression already mis-selects at evaluation time and for: is irrelevant.

How it can fail

Six specific failure shapes, each with the observable symptom that distinguishes it:

1. Stale targets in up{job="..."}. A stopped exporter still has a stale series in storage. The expression computes a 5xx rate over a window during which the denominator is zero. The rule fires on a “divide by zero omitted” outcome. Symptom: the rule fires only on instance values that no longer resolve in DNS.

2. for: shorter than one scrape interval. With scrape_interval: 15s and for: 10s, a single 5xx sample is enough to advance from pending to firing. Symptom: the alert opens and resolves inside two consecutive evaluations.

3. The expression evaluates against a counter that has reset. The rule is increase(...[5m]) on a counter that restarted within the last hour. The increase overshoots and crosses the threshold during a normal restart. Symptom: the alert correlates one-for-one with deployment windows.

4. The matcher uses job but the rename has happened. job="checkout" no longer matches. The rule never evaluates against the live series; some old recording rule is still in place and the alert fires on it. Symptom: promtool query instant returns an empty vector for the new job.

5. Alertmanager has an old silence. A silencedBy entry from a 90-day maintenance window was never reactivated and the silence has expired, but the alert history shows the alert opening and immediately resolving under a silenced marker. Symptom: the alert opens, the notification never lands.

6. The rule has no keep_firing_for and uses for: 0s. The alert flips between firing and inactive on every evaluation, generating notification churn. Symptom: the same alert produces five PagerDuty events in three minutes; Alertmanager groups them but the webhook log still shows the noise.

How to troubleshoot it

Follow the order. Layer 1 first, then 2, then 3.

1. Open the firing alert in the Alertmanager UI. Note its labels. The set of labels the alert carries is the matching set the rule evaluated.
2. Run the rule’s expression as an instant query. If the value is below the threshold and there is no recording rule in between, Layer 1 is wrong.
3. If the value is above the threshold, list the series behind it: sum by (label) (rate(...)) for each label the rule groups by. If the label set includes a synthetic probe, an old instance, a deprecated region, then Layer 2 is wrong.
4. If the value is above the threshold on the right series set, evaluate the alert history. If the alert stays pending for less than the for: duration and then resolves, Layer 3 may be wrong (and Lesson 03 applies). If the alert is in firing for a long burst, Layer 1 may still be wrong because the threshold is too tight for the load shape.
5. Form a hypothesis about the layer the rule is wrong at, edit the rule, run promtool check rules, then promtool query instant to confirm the new series set, then ship the rule change.

Security implications

A firing alert carries labels that are visible in PagerDuty, Slack, email, and any ticketing system the notification pipeline writes to. If those labels include sensitive information (an internal hostname, a customer ID, a region name not meant for disclosure), the alert is a leakage channel. In a healthy setup, alert labels are operator-readable but not customer-identifiable, and the notification templates strip secrets from annotations.

If the rule expression takes a long time to evaluate (large rate() over a long range), Prometheus can be loaded by a rule that re-evaluates every 15 seconds on a high-cardinality series set. Cardinality hygiene matters; this lesson does not treat it in depth.

Performance implications

Rule evaluation cost is bounded by the number of series the expression selects. A rule with sum by (job) over a high-cardinality metric evaluated every 15 seconds can consume single-digit percent of a Prometheus server’s rule budget. The cost is the same whether the rule fires or not.

A mis-tuned rule at Layer 2 typically has more selected series than the corrected version, so the mis-tuned version is more expensive to evaluate than the right one. For Alertmanager 0.28.x, the notification grouping machinery is single-threaded per route. A high false-positive rate inflates the grouping log size and slows the /api/v1/alerts endpoint on a busy day.

Production guidance

• Diagnose the layer first. Tune the layer that is wrong.
• Confirm the layer with promtool query instant before shipping the rule change.
• Track false-positive rate as a metric: count of alerts closed without action over count of alerts opened.
• Treat high-rate rules as candidates for quieter notification routes (chat only, not page).

Verification

You should now be able to answer:

• What are the three layers of a Prometheus alerting rule that can each produce a false positive?
• Which layer is most often the cause, and how do you recognise it from the labels on a firing alert?
• What is the first command you run against a firing rule to decide which layer to tune?
• Why is disabling or silencing a recurring false positive the wrong first move?

Quiz