Break/Fix: Alert Not Firing · Break/Fix · Observability

Reported symptoms

On the 14th, payments-api returned 5xx on roughly 22% of requests in eu-west-1 and us-east-1 for 68 minutes. The team found out from a support escalation, forty minutes after it had already recovered.

PaymentsHighErrorRate exists. It is owned, reviewed, and in version control. It did not page.

The first hour of the postmortem produced eight observations, and no two of them seem to point the same way:

The rule is loaded. /api/v1/rules returns it, with health ok, an empty lastError, and a lastEvaluation timestamp from fourteen seconds ago.
PaymentsLatencyP99 — same file, same group, same labels, same route, same receiver — fired at 09:14 during the same window and paged the same on-call engineer, who acknowledged it.
Pasting the rule expression into the Prometheus expression browser returns an empty result. Not a small number. Empty. And it still returns empty when evaluated at a timestamp inside the incident window, with all the data still in the TSDB.
The numerator of the expression, on its own, returns data.
The denominator of the expression, on its own, returns data.
The Grafana panel called “payments 5xx ratio by region” shows the 22% spike across the full 68 minutes, on both regions.
promtool check rules has passed in CI on every commit to this file, including the commit that introduced the fault.
The rule fired correctly twice earlier in the year — once in March, once in April. It has not fired since a refactor merged on 2 June that added a region breakdown.

Evidence provided

The rule as it stands, since the June refactor:

groups:
  - name: payments.rules
    interval: 30s
    rules:
      - alert: PaymentsHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{job="payments-api", code=~"5.."}[5m])
          )
          /
          sum by (service) (
            rate(http_requests_total{job="payments-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          team: payments
        annotations:
          summary: 'payments 5xx ratio above 5% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/payments/high-error-rate'

Read-only / Safeloaded, healthy, and inactive

$ curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name=="PaymentsHighErrorRate") | {state, health, lastError, evaluationTime}'

{
"state": "inactive",
"health": "ok",
"lastError": "",
"evaluationTime": 0.009
}

Illustrative output

Each half of the expression, evaluated on its own at a timestamp inside the incident window:

Read-only / Safenumerator: three series

$ promtool query instant http://prometheus:9090 'sum by (service, region) (rate(http_requests_total{job="payments-api", code=~"5.."}[5m]))'

{service="payments", region="ap-south-1"} => 0.31 @[1755156600]
{service="payments", region="eu-west-1"} => 41.7 @[1755156600]
{service="payments", region="us-east-1"} => 38.2 @[1755156600]

Illustrative output

Read-only / Safedenominator: one series

$ promtool query instant http://prometheus:9090 'sum by (service) (rate(http_requests_total{job="payments-api"}[5m]))'

{service="payments"} => 371.4 @[1755156600]

Illustrative output

Read-only / Safethe whole expression, over the incident window

$ promtool query instant http://prometheus:9090 'sum by (service, region) (rate(http_requests_total{job="payments-api", code=~"5.."}[5m])) / sum by (service) (rate(http_requests_total{job="payments-api"}[5m]))'

(no results)

Illustrative output

Two more readings that rule things out rather than in:

prometheus_rule_evaluation_failures_total for the payments.rules group is flat at zero across the incident window. The group is not failing; it is succeeding.
ALERTS_FOR_STATE{alertname="PaymentsHighErrorRate"} has produced no sample at any point in the last ten weeks. Not one pending, not one firing.

Work the evidence before reading on

Four questions. The first three are answerable from what is above; the fourth is the one that matters.

The rule is loaded, healthy, and evaluated nine milliseconds ago. Given the four states an alerting rule can be in, which one is it in, and what does that state say about the result of its expression?
Write down the label set of the numerator series and the label set of the denominator series. What does a PromQL binary operator require of two elements before it will combine them?
A rule in the same group and the same file fired and paged during the same window. Walk the six-stage pipeline from target to receiver and mark every stage that observation proves is working.
promtool check rules was green on the commit that broke this. What question does that command answer, and what question did everybody believe it was answering?

Before continuing: state the pipeline stage at which this alert fails, and name the single check that would have caught it on 2 June.

Root cause

1. The two halves cannot be matched

PromQL binary operators do not simply pair up numbers. Between two instant vectors, the default is a one-to-one match on the complete label set of each side: an element on the left combines with an element on the right only when every label name and value is the same on both.

The numerator produces elements labelled {service, region}. The denominator produces elements labelled {service}. There is no pair in the entire cartesian product whose label sets are identical, so the matcher discards everything and the division returns an empty vector. The comparison > 0.05 is then applied to nothing and also returns nothing.

This is silent by design. Vector matching has no error path for “found no partners” — an empty result is a legitimate result, and it is the same result a healthy service produces.

2. The refactor changed one line, and the defect lives between two

The 2 June commit added region to the numerator so the alert could say which region was affected. It is a one-line diff and it is correct on its own terms. The denominator was not touched, and there was no reason for a reviewer looking at the hunk to open it.

That is the shape worth remembering: the defect is not in either line. It is in the relationship between a line that changed and a line that did not.

3. Every check that ran was answering a different question

Check	What it establishes	What it does not
`promtool check rules`	the file parses and the PromQL is syntactically valid	that the expression can return anything
rule `health: "ok"`	the last evaluation completed without raising an error	that the evaluation produced a result
rule `state: "inactive"`	the expression returned no series at the last evaluation	whether that is because the service is healthy or because the expression is empty by construction
`prometheus_rule_evaluation_failures_total`	the group is not erroring	the same as above

Every one of those was green for ten weeks, and correctly so. None of them can tell a quiet rule from an impossible one, because at the level of the alert state machine there is no difference: no series, no state, no alert.

4. The sibling rule was the fastest evidence in the room

PaymentsLatencyP99 fired at 09:14 and paged. One observation, five stages eliminated:

Target scraped        -> proved by the sibling rule having data
TSDB has samples      -> proved
Rule file loaded      -> proved (same file)
Group evaluating      -> proved (same group)
Prometheus -> AM      -> proved (the page arrived)
AM route matched      -> proved (same severity and team labels)
Receiver delivered    -> proved (the on-call acknowledged)

Everything that is shared between the two rules is working. The only thing not shared is the expression. That narrows a six-stage investigation to one stage before a single command has been run, and it is available in any estate where more than one rule targets the same service.

Under the hood

It is worth being precise about why state: "inactive" is not a diagnosis on its own.

Prometheus creates one alert state machine per result series. When the expression returns three series, there are three independent state machines, each with its own dwell timer, each visible in ALERTS and ALERTS_FOR_STATE. When the expression returns nothing, there are no state machines to create, and ALERTS has no samples to expose.

So “inactive” means precisely one thing: at the last evaluation, the expression returned no series. It is the correct and expected state for a healthy service. It is also the permanent state of a rule that cannot produce a series, and the two are represented identically.

The distinguishing evidence is history, not state. A rule that has been evaluated every thirty seconds for ten weeks and has never once produced an ALERTS_FOR_STATE sample is either watching something that has genuinely never happened, or is incapable of watching anything. Both are worth a look, and the query that finds them is the same.

Resolution

Confirm the diagnosis rather than assuming it. Evaluate each half of the expression separately at the same timestamp and write down the two label sets. If they differ, the default matching rule explains the empty result completely and no further hypothesis is needed.
Decide what the denominator is supposed to mean before editing anything. A per-region ratio compares each region against its own traffic; a fleet-wide denominator compares a region against the whole estate. The two answer different questions and only one of them can detect a single-region outage.
Choose the per-region form here, grouping both sides by (service, region). The incident that started this investigation was two regions out of three, and a fleet-wide denominator would have produced a ratio around 0.2 for a service that was, in those regions, catastrophically broken.
If a fleet-wide denominator is genuinely intended somewhere else, write the join explicitly: on(service) group_left() against a denominator grouped by (service) alone, which keeps the region labels from the left-hand side and states the many-to-one relationship instead of leaving it to be inferred.
Write the unit test before applying the fix. Point it at the current rule file and run it: it must fail. A test written after the fix, that has only ever passed, is not evidence that it detects this defect.
Apply the fix, run promtool check rules, then run promtool test rules and confirm the previously failing assertion now passes.
Tell the on-call engineer before reloading. This rule has never been live. If the error ratio is above threshold at the moment of the reload, it will page as soon as the dwell elapses, and an unexplained page from a rule nobody has seen fire is its own small incident.
Reload Prometheus. The rule file is not watched; a change that is not reloaded is not applied.
Sweep the rule estate for the same shape: any arithmetic or comparison operator between two aggregations whose by clauses differ, with no explicit on() or group_left(). This is a grep-able pattern and it rarely appears alone.
Sweep separately for rules with no firing history at all, and triage each one as either untested or retirable.

Verification

The corrected expression, evaluated at timestamps inside the 68-minute window, returns values above the threshold for the two affected regions. The incident data is still in the TSDB, which makes this the strongest verification available: a direct demonstration that the rule would have fired on the real failure.
The unit test fails against the pre-fix rule and passes against the corrected one. Keep both runs in the change record; the failing run is the part that proves the test detects something.
The rule fires end to end in a non-production replica, and the notification reaches the receiver. The first firing in this rule lifetime should not be during a real incident.
After the reload, ALERTS_FOR_STATE begins producing samples for the alert as the expression moves through its states in normal operation. A rule that still produces no samples has not been fixed.
The sibling rules in the same group still behave as before. A change to one expression should not alter another, and confirming that is cheap.
The estate sweep returns a list, and every entry on it has an owner and a decision. An unread list is not a completed step.
Quiet is not accepted as evidence anywhere in this checklist. Every item above asks for something to happen, because the symptom under investigation was nothing happening.

Prevention

Require a firing assertion in every alert rule test. A test that only asserts exp_alerts: [] under quiet input passes against a rule that can never fire. The assertion that matters is the one that drives synthetic data past the threshold and expects an alert after the dwell.
Make the binary-operator rule explicit in review. Two aggregations joined by an arithmetic or comparison operator must have identical by clauses, or an explicit on() with group_left(). Anything else is relying on two lines agreeing by habit.
Read a diff against the lines it did not change. A one-line change to one side of an expression is a change to the expression, and reviewing the hunk alone cannot see that.
Track rules that have never fired. The data is already there in the alert history. A critical rule with no firing history after a quarter is either untested or unnecessary, and both need a decision.
Stop treating a dashboard panel as corroboration. The panel and the rule are different queries unless one is generated from the other. Here the panel was right for ten weeks while the rule was empty, and the agreement everybody assumed existed had been broken by a commit that touched neither of them together.
Start every “should have fired” investigation with a sibling. Find something that did fire and shares as much of the pipeline as possible. It is the cheapest way to eliminate five stages, and it costs one look at the notification history.
Exercise critical alerts on a cadence. A quarterly drill that drives each paging rule to fire in a replica converts a rule from a hypothesis into a control, and it catches this defect class along with expired credentials and stale routes.

What if the refactor had added region to the denominator instead of the numerator?

Exactly the same outcome: an empty vector, a permanently inactive rule, and ten weeks of silence.

The failure is symmetric because the matching rule is symmetric. It does not care which side gained a label; it cares only that the two label sets are no longer identical. That symmetry is why the defect cannot be found by reading either line and judging it — both lines are individually reasonable, and a reviewer asked “is this line correct” will say yes about whichever one they are shown.

The variant that is genuinely harder is a partial match. If the numerator had gained region and the denominator had gained availability_zone, the result would still be empty, but the diff would show two changed lines and look symmetric, which is more convincing and equally wrong. The variant that is harder still is a label that matches on some series and not others: then the expression returns some series, the rule fires sometimes, and the gap in coverage is invisible because the alert has a firing history and therefore looks proven.

The check that survives all three variants is the same one: assert, under synthetic input, that the rule produces an alert. Reading expressions catches the obvious cases; running them catches the rest.

Why this matters

Sixty-eight minutes of a payments service returning 5xx to a fifth of its traffic, with a paging rule loaded, healthy, evaluated twice a minute, and structurally unable to say so.

Two things are worth carrying out of this. The first is mechanical: a binary operator in PromQL is a join, joins need matching keys, and when the keys stop matching the result is not an error but an absence. Absence is the one result an alerting system cannot distinguish from good news.

The second is about what the green checks were actually claiming. promtool check rules says the file parses. health: "ok" says the evaluator did not crash. Neither says the rule means what you think, and a team that reads them as a safety net has a safety net made of two statements about syntax.

The rule that can prove it fires is the only one you have. Everything else is a rule you have not seen fail yet.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The two halves cannot be matched

2. The refactor changed one line, and the defect lives between two

3. Every check that ran was answering a different question

4. The sibling rule was the fastest evidence in the room

Resolution

Verification

Prevention