Runbook: Investigate an Alert Not Firing

The cost of this failure is asymmetric. A false positive wakes someone who could have slept. An alert that does not fire leaves someone asleep through a real incident, and the outage lasts from the moment the condition appeared to the moment somebody found out by other means — usually a customer.

That asymmetry is why this runbook is ordered rather than exhaustive. There are six stages between a metric and a pager, each with a check that takes seconds. The discipline is to run them in order and stop at the first one that fails.

Before you start: is this actually a non-firing alert?

Three things must all be true, and confirming them takes a minute:

1. The condition is true right now. Not an hour ago. A rule that stopped firing because the problem stopped is correct.
2. A rule exists whose purpose is this condition. If there is no rule, this is a rule gap: a design conversation for the morning, not a pipeline to debug.
3. No notification reached the receiver. Check the receiver, not the team’s memory of it.

The pipeline, and why the order is not negotiable

target /metrics  ->  TSDB  ->  rule evaluator  ->  alert state
                                                        |
                                                        v
    receiver   <-  route match  <-  silence / inhibit / group

Each stage has a faster check than the one after it. Reaching for amtool before you have run the rule’s expression means ten minutes spent proving a route is correct while the expression it depends on returns nothing.

Roughly half of real cases end at stage 2 — the expression returns no series. That is not a guess about your estate; it is where label drift accumulates fastest, because every exporter upgrade and every instrumentation change is an opportunity for a matcher to stop matching.

Stage 1 — Is the rule loaded, and is it healthy?

promtool check rules /etc/prometheus/rules/checkout.yml

PROM=http://localhost:9090
curl -sf "$PROM/api/v1/rules" \
| jq -r '.data.groups[] | .file as $f | .rules[]
    | select(.name == "CheckoutHighErrorRate")
    | [$f, .name, .health, .lastError] | @tsv'

Two distinct questions live here. promtool check rules answers “does this file parse”. The API answers “is this rule actually loaded in the running server”, which is a different thing entirely — a file that was never deployed, or a rule_files glob that stopped matching, parses perfectly and is loaded nowhere.

A rule can also be loaded and still not evaluating. Two metrics answer that:

# expressions that error at evaluation time
prometheus_rule_evaluation_failures_total

# groups that could not finish inside their interval and skipped a tick
prometheus_rule_group_iterations_missed_total

A missed iteration is the quiet one. The rule is present, healthy in the UI, and simply not being run often enough — usually because an expensive expression in the same group is eating the interval.

Stage 2 — Does the expression return a series?

This is where you should expect to spend your time.

PROM=http://localhost:9090

curl -sfG "$PROM/api/v1/query" \
--data-urlencode 'query=sum by (service, region) (rate(http_requests_total{job="checkout-svc",code=~"5.."}[5m]))' \
| jq '.data.result | length'

Copy the expression out of the rule file rather than retyping it. The whole point is to test the expression that is loaded, not the one you believe is loaded.

If the result is empty, the rule sits inactive forever regardless of what the system is doing. Now find out why by walking the matchers outward, dropping one at a time:

PROM=http://localhost:9090

curl -sfG "$PROM/api/v1/query" \
--data-urlencode 'query=count by (job, code, environment) (http_requests_total)' \
| jq -r '.data.result[] | [.metric.job, .metric.code, .metric.environment] | @tsv'

Then compare what came back with what the rule asks for. The four shapes, in rough order of frequency:

The canary case is the one that costs the most, because the alert works. It fires for the population it was written against and silently misses the new one. Nobody notices until the missed population is the one that breaks.

Stage 3 — Has the dwell elapsed?

PROM=http://localhost:9090

curl -sfG "$PROM/api/v1/query" \
--data-urlencode 'query=ALERTS_FOR_STATE{alertname="CheckoutHighErrorRate"}' \
| jq -r '.data.result[] | [.metric.instance, .value[1]] | @tsv'

0 is inactive, 1 is pending, 2 is firing. The state is per series, not per rule: a rule returning three series runs three independent state machines, and one of them can be stuck while the others are fine.

If series reach pending and never advance, the condition is ending before the dwell does. Total time to fire is the sum of four things:

time_to_fire = scrape_interval + rule_interval + for_dwell + group_wait

With a 15s scrape, a 30s rule interval, for: 5m and a 10s group_wait, the alert lands between 5m10s and 5m55s after the condition begins. If the condition typically lasts ninety seconds, no amount of debugging will make that alert fire — the dwell is simply wrong for the thing it watches.

Stage 4 — Did the alert reach Alertmanager?

PROM=http://localhost:9090

curl -sf "$PROM/api/v1/alertmanagers" | jq '.data'

curl -sfG "$PROM/api/v1/query" \
--data-urlencode 'query=prometheus_notifications_errors_total'

amtool alert query --alertmanager.url=http://alertmanager:9093

/api/v1/alertmanagers tells you whether Prometheus has an Alertmanager target at all. An empty list is a decisive answer: the alerting: block in prometheus.yml is wrong or its discovery resolves to nothing, and every alert in the estate is affected, not only this one. That changes the severity of what you are handling.

If the alert is firing in Prometheus and absent from amtool alert query, the failure is in that hop. If it is present, move on — stages 5 and 6 are Alertmanager’s business.

Stage 5 — Did the route match?

amtool config routes test \
--config.file=/etc/alertmanager/alertmanager.yml \
alertname=CheckoutHighErrorRate \
severity=page \
team=checkout

Feed it the labels Prometheus actually sent, copied from amtool alert query — not the labels the rule file says it sets. Those can differ: external labels are added, and relabelling can rewrite them on the way out.

The classic defect is a matcher that is almost right. A route matching severity = "pages" against a rule emitting severity: page matches nothing, so the alert falls through to the default receiver. It was delivered — just to a channel nobody watches at 03:00. This looks identical to “no alert” from the on-call’s side, and it is why the check is a routing test rather than a search of the pager history.

Stage 6 — Did the receiver accept it?

# failures per integration - the label names the receiver type
alertmanager_notifications_failed_total

# for comparison: the attempt count
alertmanager_notifications_total

Read the Alertmanager log alongside them. A failing integration answers with a status code, and the status code is the diagnosis: 401 after a credential rotation, 429 from a rate-limited API, a connection error from an outbound path that a firewall change closed. Check the receiver’s own status page before assuming the fault is local.

Holding, when the fix is not immediate

If the condition is live and the pipeline is broken, detection for that failure mode does not exist. That is a state you may have to accept for a while, but not one to leave implicit.

Name a person to watch the condition by hand, name what they are watching — the dashboard panel, the query, the customer channel — and name the time the arrangement ends. “Someone will keep an eye on it” is not a control. A named human, a named signal and an end time is.

Rollback

Everything up to stage 5 is read-only. What needs undoing is what you changed to investigate:

git revert --no-edit HEAD
promtool check rules /etc/prometheus/rules/checkout.yml
curl -sf -X POST "http://localhost:9090/-/reload"

A rejected reload changes nothing — the previously loaded rules keep running. That is a safety property, not a reason to restart the server.

Close it with a test, not with a fix

The defect that caused this is a rule-file defect that loaded cleanly and evaluated quietly. Nothing will catch the next one except a test that encodes the case:

# test/checkout_test.yml
rule_files:
  - ../rules/checkout.yml
evaluation_interval: 30s

tests:
  - interval: 30s
    input_series:
      - series: 'http_requests_total{job="checkout-svc-canary",code="500"}'
        values: '0+10x20'
    alert_rule_test:
      - eval_time: 6m
        alertname: CheckoutHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              team: checkout

Run it with promtool test rules test/checkout_test.yml. The test must fail against the old rule before you trust it — a test written after the fix that passes immediately has proved nothing.

Escalation

• The condition is a live customer-visible failure: escalate now, in parallel. Two problems, two people.
• The fix is a threshold, a dwell, or which population should be covered: that belongs to the service owner.
• Stage 4 failed: platform team, and treat it as estate-wide.
• Stage 6 shows repeated integration errors: the receiver’s owner, plus its status page.
• Thirty minutes with no stage identified: re-examine the assumption that the condition is true and that a rule covers it.