Alert Failure Anatomy · Observability

A scheduled checkout-svc health check should fail at 03:00 and page the on-call. The page does not arrive. The on-call is asleep, unaware. At 08:30 a customer reports that checkout has been broken since 03:00. A production alert existed, passed review, was merged, was deployed, and yet failed to perform its single job. The next hour belongs to the team that can name the layer at which it failed in five minutes or less.

This lesson is the map of that pipeline and the diagnostic order that makes five-minute diagnosis reliable. The remaining five lessons in this module zoom into each layer.

What a non-firing alert is

A non-firing alert in production terms is the absence of a notification on a system state that the alerting contract requires the operator to be told about. Three conditions must all hold:

The underlying condition is true at the moment of investigation (the metric is wrong, the dependency is down, the queue is stalled).
The team has a rule whose purpose is to fire on that condition.
No notification reached the receiver that the rule should have routed to.

Any one of those three failing means it is not yet a non-firing alert. A rule with no condition present is correct behaviour. A rule with a condition present but no rule in place is a rule gap, not a rule failure. This lesson assumes conditions one and two hold; the diagnostic works the pipeline backwards from there.

A non-firing alert is not the same as a silenced alert or a suppressed alert. Silenced and suppressed alerts fire in the state machine but never reach the operator by policy. A non-firing alert never reaches the firing state at all.

Why a sysadmin cares

The cost of a non-firing alert is asymmetric with the cost of a false positive. A false positive wakes someone up who could have been asleep. A non-firing alert leaves someone asleep during a real incident. The first cost is a productivity tax; the second cost is an outage that lasts from the moment the condition appeared to the moment the team found out by other means.

Two failure shapes repeat:

The slow outage. A condition persists for hours. Customers notice before the team does. Mean-time-to-detect is minutes, but mean-time-to-know is the customer-ticket queue. The postmortem names the missing page.
The silent edge case. A condition exists on exactly one host, one region, or one tenant. The rule exists, fires for the population it was written for, and silently misses this one case. The team finds out when the customer reports it.

The fix in both cases is the same: a diagnostic order short enough to run inside a ten-minute window, with the cheapest checks at the top.

How it works

A Prometheus alert is a flow through a six-stage pipeline. Each stage has a state and a transition condition. A non-firing alert is a pipeline that runs to completion but stops somewhere before the operator.

+----------+    +----------+    +-----------+    +-----------+
| Target   |--->| Prometheus|--->| Rule      |--->| Alert     |
| /metrics |    | TSDB     |    | evaluator |    | state     |
+----------+    +----------+    +-----------+    +-----------+
  scrape         1m retention    expr -> for:    inactive ->
  reachable      block storage   series          pending ->
                                                firing
                                                   |
                                                   v
+----------+    +----------+    +-----------+
| Receiver | <--| AM       | <--| AM        |
| webhook  |    | route    |    | grouping  |
| PD/Slack |    | match    |    | silence   |
+----------+    +----------+    +-----------+
  integration   label match     inhibition
  returns 2xx   against route   final group

Stage 1: Scrape. Prometheus or an agent pulls /metrics from the target. The target must be reachable, the endpoint must respond, and the response must be parseable. A failed scrape leaves no new samples in the TSDB.

Stage 2: TSDB. Samples land in the block storage, indexed by metric name and label set. A scrape that succeeded at 03:00 and a fail at 03:01 leaves a gap in the series.

Stage 3: Rule evaluator. On every evaluation_interval, the loaded rule expressions run. Each rule returns a vector. The for: timer advances only for series that remain in the result.

Stage 4: Alert state. Per-series state transitions inactive to pending to firing. A series in firing is published to the notification log.

Stage 5: AM routing. Alertmanager receives the firing alert, applies silences, inhibitions, group-wait, group-interval, and route matching. A matched route produces a notification per receiver.

Stage 6: Receiver. The receiver performs an integration call: HTTP POST to a webhook, HTTP POST to PagerDuty Events API, SMTP to an email relay. A successful HTTP 2xx returns delivery.

A non-firing alert fails at one of these six stages. The diagnostic walks them in order from cheapest to most expensive.

The diagnostic order

Order matters because each layer has a faster validation step than the next, and because acting on the wrong layer wastes time and risks silencing a real alert to mask a different problem.

+----------------------------------------+
| 1. Rule loaded?                         |
|    promtool check rules                 |
|    promtool test rules                  |
|    ALERTS{alertname=...}                |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 2. Expression returns a series?        |
|    promtool query instant 'expr'        |
|    /api/v1/query?query=expr            |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 3. for: elapsed?                        |
|    ALERTS_FOR_STATE{alertname=...}     |
|    inspect alert history                |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 4. Firing alert reached AM?            |
|    amtool check-config                  |
|    AM /api/v1/alerts                    |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 5. Route matched?                       |
|    amtool config routes test            |
|    AM debug routing                     |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 6. Receiver delivered?                  |
|    amtool check-config receiver         |
|    AM logs (notify success / errors)    |
|    curl against webhook URL             |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| Find the first stage that fails.        |
+----------------------------------------+

Step 1 is the cheapest: did the rule load? promtool check rules parses the file and reports syntax errors with line numbers. If a rule syntax breaks at load, every rule in the file is dropped, not just the broken one. A typo in one alert rules out the whole file.

Step 2 is the expression: does the query return any series? A rule that parses fine and produces no vector at evaluation time sits in inactive forever. Label drift, renamed metrics, and deprecated collectors all land here.

Step 3 is the timer. Even with a vector in hand, an alert stays pending for for:. If for: 30m was set against a rule that should page within five minutes, the alert is not broken; it is wrong.

Step 4 is the AM handoff. Prometheus writes the firing alert to its notification log and AM consumes it. Connection issues, the wrong AM target, a SIGTERM on the AM pod, all break here.

Step 5 is the route. AM receives the alert, applies group logic, matches the labels against route.match and route.matchers, and produces a notification target. A missing matcher sends the alert to the default route; a wrong matcher sends it nowhere.

Step 6 is the integration. The receiver performs the out-of-band call. Auth failure, DNS resolution, TLS handshake, rate-limit response from PagerDuty, SMTP relay down, all break here.

The most common cause

In roughly half of the production “alert should fire but does not” cases a team investigates, the cause is at Step 2: the expression returns no series. Specifically:

A label matcher refers to a label whose value drifted. The metric exists, but the combination the rule expects does not.
The metric was renamed or replaced in a newer exporter version, and the rule file still targets the old name.
The recording rule that fed the alert went away in a config edit, and the alert rule still references it by name.
The sum by or avg by clause omits a label the team expects to aggregate by, and the resulting vector matches the wrong series.

These are rule-file defects that load cleanly and evaluate quietly. They are invisible until the condition actually appears and the team expects the page that does not come.

The other half of the time, the cause is evenly split across the remaining five layers. Layer 2 is dominant only because it is the layer where label drift accumulates fastest.

Under the hood

When a rule loads and evaluates, Prometheus creates a virtual series for every label combination the expression returns. Each of those series carries its own state. The ALERTS metric and the ALERTS_FOR_STATE metric expose that state as data. A query against ALERTS{alertname="CheckoutHighErrorRate"} returns one sample per series with alertstate set to pending or firing. A query against ALERTS_FOR_STATE{...} returns one sample per series with the numeric state.

This matters for the diagnostic. When a rule is healthy, the query returns series in pending or firing for the population the rule was written against. When the rule is broken, the query returns nothing, or returns series with labels the team does not recognise as production.

The flow from Prometheus to Alertmanager is a single HTTP POST on each evaluation in which a series transitions between states. Prometheus writes the alert to its notification log (the WAL plus a notifications directory) and AM consumes it. AM is the authority on grouping, silencing, and routing from that point. If the alert has reached AM, it appears in /api/v1/alerts on the AM HTTP endpoint.

The contract between Prometheus and Alertmanager is a labelled JSON payload, not a query result. Each alert carries {alertname, severity, team, ...} plus the metric value and annotations. The label set is the entire routing input. If Prometheus drops a label at the rule (or relabel), AM cannot match on it.

How to configure it

The configuration that protects a non-firing alert is mostly around the rule file. Real annotated rules with diagnostic hooks:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"5..",
                environment="production",
              }[5m]
            )
          )
          /
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"2..|3..|4..|5..",
                environment="production",
              }[5m]
            )
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout error rate above 5% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The lines worth highlighting for the diagnostic:

job="checkout-svc" is a label matcher. If the actual job label drifts to checkout-svc-canary for the canary fleet, the rule never sees the canary. Step 2 fails.
for: 5m is the dwell timer. With interval: 30s the rule evaluates every 30 seconds; advancing from pending to firing requires ten consecutive evaluations with the same result series. Step 3 fails if the dwell timer is too long.
severity: page and team: checkout are the labels AM routes against. If the rule file sets severity: warn by accident, AM never matches the page route. Step 5 fails.

The Alertmanager configuration that protects the routing side:

route:
  receiver: default
  group_by: ['alertname', 'region']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "page"
        - team = "checkout"
      receiver: checkout-pagerduty
      continue: false
receivers:
  - name: default
    webhook_configs:
      - url: 'https://hooks.example.com/default'
  - name: checkout-pagerduty
    pagerduty_configs:
      - routing_key: 'PD_INTEGRATION_KEY_REDACTED'
        send_resolved: true

The matchers: block inside the child route is what AM evaluates against the labels Prometheus sends. A misspelt severity = "pages" (plural) matches nothing; every page-worthy alert falls through to default.

How to validate it

Six checks, one per pipeline stage. Run them in order.

Stage 1: rule is loaded.

promtool check rules /etc/prometheus/rules/checkout.yml

Successful output:

SUCCESS: rule files validated; 14 rules found, 0 errors

A parse error includes the line number:

err: yaml: line 42: did not find expected key
SUCCESS: rule files validated; 13 rules found, 1 errors

Stage 2: expression returns a series.

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'sum by (service, region) (
     rate(http_requests_total{job="checkout-svc",code=~"5..",environment="production"}[5m])
   )'

If the result is [], the expression returns no series; the rule will sit inactive forever regardless of conditions.

Stage 3: alert state.

curl -s http://prometheus:9090/api/v1/query?query=ALERTS_FOR_STATE \
  | jq '.data.result[] | {alertname: .metric.alertname, state: .value[1]}'

1 means pending. 2 means firing. 0 means inactive.

Stage 4: AM received the alert.

amtool check-config /etc/alertmanager/alertmanager.yml

Successful output:

Checking 'alertmanager.yml' SUCCESS
Found 2 routes, 3 receivers, 1 inhibit rules, 1 templates

Then on the AM side:

curl -s http://alertmanager:9093/api/v1/alerts \
  | jq '.data[] | select(.labels.alertname=="CheckoutHighErrorRate")'

Stage 5: route matched.

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --alertmanager.url=http://alertmanager:9093 \
  --source.alertfile=/tmp/sample-alert.json

The output lists which receiver received the alert for the label set in the sample.

Stage 6: receiver delivered.

journalctl -u alertmanager --since "10 minutes ago" \
  | grep -E "notify|webhook|pagerduty"

A successful notification ends with notify success and the HTTP 2xx code returned by the receiver integration.

How it can fail

Six failure shapes, each tied to a stage in the pipeline:

Rule file fails to load. A single YAML indentation error in a rule file drops every rule in that file, not just the broken one. Symptom: ALERTS{alertname="X"} is empty while the rule file exists at the path; promtool check rules reports the parse error.
Expression returns no vector at evaluation time. Label drift, renamed metric, deprecated collector. The rule evaluates, finds zero series, advances no state. Symptom: promtool query instant 'expr' returns [] and the underlying metric exists in /api/v1/series.
for: longer than the natural duration of the condition. The condition exists for ninety seconds; for: is set to 30m. The rule goes pending and resolves before for: elapses. Symptom: ALERTS_FOR_STATE shows 1 (pending) briefly followed by no sample; the alert has never gone 2 (firing).
AM and Prometheus URL are wrong. Prometheus is configured to push to http://alertmanager:9093 but AM is bound to 0.0.0.0:9093 only on the loopback interface. Symptom: Prometheus logs failed to send alerts; AM logs are silent.
Route matcher uses a renamed label. The rule emits severity: page, but the route matcher is severity = "pages" (plural). AM matches the default route, which posts to a non-paging integration. Symptom: an alert in /api/v1/alerts with severity: page reaches the wrong receiver; the runbook never opens.
Receiver integration credential expired. The PagerDuty Events API integration key was rotated but routing_key in AM is the old one. Symptom: AM logs notify attempt X failed: 401 from events.pagerduty.com; nflog shows the alert as errorState: "error".

How to troubleshoot it

Follow the six steps. Do not skip.

Step 1, rule loaded. promtool check rules. If the parse fails, fix the file. If it passes, move on.
Step 2, expression returns a series. If the expression returns no vector but the underlying metric exists, the label matcher is wrong. Edit the rule file to match the label set you find with /api/v1/series on the metric.
Step 3, for: elapsed. Inspect the per-series state. If the rule is pending past the for: duration without advancing, the rule file’s for: may be mis-clocked against the actual scrape interval. Reduce for: to something just longer than the scrape interval plus one evaluation, and re-validate.
Step 4, AM handoff. If AM shows no entry in /api/v1/alerts despite a firing sample in Prometheus, the webhook from Prometheus to AM is broken. Check the alerting: block in prometheus.yml and the AM log for tsdb rejects of the notifications directory.
Step 5, route matched. If AM received the alert but routed to the default receiver, the route matcher is wrong. Use amtool config routes test with a sample label set to reproduce.
Step 6, receiver delivered. If AM delivered to the right receiver but the integration is down, AM logs the failure with the HTTP status returned. Rotate credentials, confirm the receiver’s API status page, and use the retry_on_not_ready knob on pagerduty_configs for known transient states.

Security implications

A non-firing alert can be a security signal masking itself. If the rule that should fire on authentication anomalies is also broken, an attacker making silent progress gets hours of lead time before any other signal catches them. Run the diagnostic for security-critical alerts with the same discipline as availability alerts.

The receiver side of the pipeline handles credentials (webhook signing keys, PagerDuty integration keys, SMTP passwords). AM stores these in alertmanager.yml. File permissions on that file must restrict read to the AM process account. Rotation of these credentials is part of the alert delivery contract, not a separate concern.

Performance implications

Each rule evaluates on its interval regardless of whether it fires. A rule that loads but returns no vector still costs one PromQL evaluation per interval. The cost scales with the number of series the expression selects, not with the number of firing alerts. A stale rule that was never deleted after the recording rule it depended on was removed continues to evaluate every minute forever.

AM’s grouping, silencing, and routing are CPU-bound per incoming firing alert. A bad route match that fans the alert to all receivers amplifies this cost. The right fix is at Stage 5 (route matcher) not at AM’s worker count.

Production guidance

Pin every rule to a stable alertname and a for: value that is at least one evaluation interval longer than the natural minimum breach duration.
Track which rules never fire. A rule that has not fired in ninety days is either a candidate for retirement or a sign that the expression returns nothing; run promtool query instant against it on a schedule and store the result.
Run the diagnostic order for every “alert should fire but does not” investigation. The cheapest check is the first one; run it.
Treat the rule file as code: under version control, reviewed, tested with promtool test rules, and reproducible from commit SHA.

Verification

You should now be able to answer:

What are the six stages of the Prometheus-to-receiver pipeline?
In what order should you diagnose a “should fire but does not” alert, and why does the order matter?
Which stage accounts for the largest share of non-firing alerts, and what is its symptom?
How do you confirm that AM received a firing alert, independent of the receiver delivery result?