Break/Fix: Alert Firing Continuously

Reported symptoms

You take the on-call handset on a Monday morning. The handover list has one item on it, and it has been on the list for three shifts.

CheckoutHighErrorRate is firing. It started firing on the 6th. It is now the 17th. It has not resolved once in between.

The notes left by the three engineers who held the handset before you:

• 7th — “Threshold was 5%, looked too tight. Raised it to 8%. Still firing. Silenced 24h.”
• 11th — “Dashboards are green all week, cannot reproduce. Silenced 24h.”
• 14th — “for: was 5m, looks short for this service. Raised it to 30m. Alert came back thirty minutes later. Silenced 24h.”

Alongside that, four facts that nobody has yet connected:

1. The checkout 5xx ratio panel on the service dashboard has been at or below 0.4% for the whole eleven days. Every engineer who looked found nothing.
2. CheckoutLatencyP99 — a different rule, a different expression, a different threshold — started firing on the 6th as well, and has also never resolved.
3. On the 12th, a bad deploy broke checkout for forty minutes. Nobody was paged.
4. Each silence expires after 24 hours and the page arrives again within about a minute of expiry.

Evidence provided

$ curl -s http://prometheus:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="CheckoutHighErrorRate") | {state, activeAt, labels}'

{
"state": "firing",
"activeAt": "2026-08-06T09:41:30.000Z",
"labels": {
  "alertname": "CheckoutHighErrorRate",
  "service": "checkout",
  "region": "synthetic-1",
  "severity": "page",
  "team": "checkout"
}
}

$ curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS_FOR_STATE%7Balertname%3D%22CheckoutHighErrorRate%22%7D' | jq '.data.result[] | {region: .metric.region, state: .metric.alertstate, seconds: .value[1]}'

{
"region": "synthetic-1",
"state": "firing",
"seconds": "951420"
}

The rule itself, as it stands in git this morning, after both edits:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{job="checkout-svc", code=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{job="checkout-svc"}[5m])
          )
          > 0.08
        for: 30m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout 5xx ratio above threshold in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The expression run as an instant query, with the threshold comparison stripped off so the actual values are visible:

$ promtool query instant http://prometheus:9090 'sum by (service, region) (rate(http_requests_total{job="checkout-svc", code=~"5.."}[5m])) / sum by (service, region) (rate(http_requests_total{job="checkout-svc"}[5m]))'

{service="checkout", region="ap-south-1"} => 0.0035 @[1755424800]
{service="checkout", region="eu-west-1"} => 0.0038 @[1755424800]
{service="checkout", region="us-east-1"} => 0.0041 @[1755424800]
{service="checkout", region="synthetic-1"} => 0.41 @[1755424800]

Two more pieces sit in version control and in Alertmanager rather than in Prometheus:

• The scrape configuration gained a relabel_configs entry on the 6th, in a change titled “unify checkout target labelling”, that rewrites job to checkout-svc for the load-generator targets.
• The Grafana dashboard’s region template variable is populated by a label-values query restricted to environment="production".
• All three silences match on alertname="CheckoutHighErrorRate" and nothing else. The comment field is empty on all three.

Work the evidence before reading on

The alert is not lying, and neither is the dashboard. Both are describing the system accurately. They are describing different parts of it.

1. Compare the label set on the firing alert against the label sets in the instant-query output. Which of the four series is the one that is firing, and would the dashboard show it?
2. The threshold moved from 0.05 to 0.08 and the alert did not pause for a single evaluation. What does that tell you about the value of the firing series, without needing to look it up?
3. for: moved from 5m to 30m and the alert returned after exactly 30 minutes. What does that tell you about whether the breach is continuous or intermittent?
4. Two independent rules against the same service began firing on the same day. What kind of change can do that to two rules at once, given that neither rule file was edited?

Before continuing: name the layer of the rule that is wrong — threshold, series selection, or dwell time — and state the single query result that settles it.

Root cause

1. The rule selects a population it was never calibrated against

The expression constrains two things: job="checkout-svc" and the status code. That is the whole of its selection. There is no environment matcher, no exclusion for canary, shadow or synthetic traffic, and nothing else that would bound the input to production.

On the 6th, the relabel change gave the load-generator targets the job value checkout-svc. That fleet exists to drive a fixed, deliberate error rate against the checkout API so that the retry and circuit-breaker paths are exercised continuously. It is doing exactly what it was built to do. From that evaluation onward it was inside the rule’s input set.

The aggregation is what turns this from a dilution into an alert. sum by (service, region) groups the input, and the load generator carries region="synthetic-1", so it does not average into the production numbers — it produces a fourth series of its own, at 0.41.

That series has been above every threshold the rule has ever carried, on every evaluation, since the 6th.

2. Why the fire is continuous rather than flapping

for: gates the transition from pending to firing, and the timer resets on any evaluation where the expression returns no series for that label set. The load generator runs on a schedule that never stops. There has been no empty evaluation for synthetic-1 in eleven days, so there has been no reset. The alert entered firing once and has had no path back to inactive.

This is the difference between the two shapes the phrase “alert firing continuously” covers. A flapping alert has an oscillating input and needs keep_firing_for or a longer for:. A permanently firing alert has a permanently true condition, and the only questions worth asking are whether the condition should be true and whether the rule should be looking at it.

3. Why the two attempted fixes could not have worked

Neither number was ever close. Raising the threshold to 0.08 against a series at 0.41 is not a small miss, it is a change with no reachable effect short of setting the threshold above the level at which a real incident lives. Lengthening for: against a breach with no gaps in it buys exactly the length of the extension, once.

Both edits are the second-most-common shape of a false-positive investigation: tuning the layer below the one that is wrong. The threshold and the dwell time are Layer 1 and Layer 3. The defect is at Layer 2, the series the expression selects, and no amount of Layer 1 or Layer 3 tuning reaches it.

4. The silences did more damage than the alert did

Each of the three silences matched on alertname alone. A silence that broad does not suppress the noisy series; it suppresses the alert name, across every label set the rule can produce, including the three production regions.

On the 12th, checkout genuinely broke. The production series crossed the threshold, the rule advanced to firing for region="eu-west-1" — and Alertmanager suppressed the notification, because a silence covering the alert name was active at the time. That is the forty minutes with no page.

Resolution

1. Stop tuning. Both remaining knobs have been tried and neither moved the alert, which is itself the evidence that the fault is not in either of them. Do not raise the threshold a third time.
2. Establish the intended population explicitly. Run the stripped expression, list every series it returns, and decide for each one whether the team meant to alert on it. Here that is three production regions in and one synthetic population out.
3. Add the constraint positively, on both halves of the ratio. An environment="production" matcher is the fix; exclusions for known canary, shadow and synthetic labels harden it against the next population that arrives.
4. Test the new expression as an instant query before it goes anywhere near a reload, and read the result. It must return three series, not zero. A rule change that silences the alert by selecting nothing is the same defect inverted and it looks identical in the alert list.
5. Apply the same fix to CheckoutLatencyP99 in the same commit. It has the same missing matcher and started firing for the same reason; leaving it means repeating this investigation in a fortnight.
6. Run promtool check rules on the changed file, then reload Prometheus. Prometheus does not watch the filesystem; a change that is not reloaded is not applied.
7. Decide separately whether the load generator should carry job="checkout-svc" at all. If the relabel was a mistake, fix the scrape config as well - but not instead. The rule was already too broad on the 5th; the relabel only made that visible.
8. Expire the silences by hand rather than waiting for them to lapse, and confirm the alert list is empty afterwards for the right reason.
9. Write the investigation note: the layer that was wrong, the change that fixed it, and the two changes that were inert and why.

Verification

1. The alert resolves, and the resolution reaches the receiver. Watch for the resolved notification rather than for the alert disappearing from /api/v1/alerts - the second can happen because the rule now selects nothing.
2. The new expression returns one series per production region, with values in the low thousandths. This is the check that distinguishes a fixed rule from a rule that has been quietly emptied.
3. The rule can still fire. Drive a production region above the threshold in a promtool test rules unit test and assert a firing alert after the for: dwell. A rule whose only demonstrated behaviour is not firing has not been verified.
4. The dashboard panel and the rule expression now select the same population. Compare the label sets returned by each; they should agree, which they did not before.
5. CheckoutLatencyP99 has also resolved. If it has not, the shared cause was not the only cause and the investigation is not finished.
6. No silence matching this alert name remains active. List the silences explicitly rather than assuming the expired ones lapsed.
7. The ALERTS_FOR_STATE series for the alert is gone, or belongs to a production region with a dwell measured in minutes rather than in days.

Prevention

• State the intended population in every rule. An environment matcher is the cheapest one and the most often missing. A rule that bounds only job is trusting a label it does not own.
• Treat a scrape-config or relabel change as a rule change. Nobody edited either rule file on the 6th, and both rules changed behaviour. Any change that rewrites job or instance should be reviewed against the rules that select on those labels.
• Record the basis for a threshold in an annotation. A number with a recorded basis can be defended or revised. A number with no basis gets raised every time somebody is tired, which is how thresholds drift upward until they sit above the incident band.
• Read an inert change as a diagnosis. A tuning change that produces no observable effect is not a failed fix, it is a successful experiment: it rules out the layer you adjusted. Two inert changes in a row should have redirected this investigation on the 14th.
• Bound every silence and comment it. Owner, reason, and an expiry tied to the work it covers. Scope it to the labels of the series you are suppressing, never to alertname alone.
• Watch for alerts whose labels nobody recognises. A label value that no engineer on the owning team can name is the strongest single indicator that a rule is selecting outside its intended population, and it is visible on the notification itself.
• Query the alert estate for long-lived firing states. A rule that has been firing for days without resolving is either an unattended incident or a broken rule, and both deserve a look. The dwell is already exposed by ALERTS_FOR_STATE.