The Alert Rule Anatomy

A 03:00 page fires: OrdersApiHighErrorRate. The on-call opens the runbook, follows the dashboard link, and within four minutes has narrowed the fault to a single region of orders-api. That outcome is not luck. The rule that fired had a stable name, a sensible for:, the labels the Alertmanager route needs, and annotations that pointed straight at the runbook and the dashboard. Every one of those choices was made in the YAML of the rule file. This lesson is what those choices look like and why they matter.

What it is

An alert rule in Prometheus 2.55 is a YAML record under rule_files that pairs a PromQL expression with the metadata required to make a firing alert actionable. The full top-level shape is:

groups:
  - name: <group_name>
    interval: <duration>           # optional; defaults to global evaluation_interval
    limit: <int>                   # optional; max series per group evaluation
    rules:
      - alert: <AlertName>
        expr: <PromQL>
        for: <duration>            # optional; default 0s
        keep_firing_for: <duration> # optional; Prometheus 2.42+
        labels:
          <key>: <value>
        annotations:
          <key>: <value>            # Go-template string allowed

The keys have distinct jobs. alert is the unique name across all loaded rules. expr is the query that produces the result series. for is the dwell time before pending becomes firing. keep_firing_for is the optional post-fire dwell time before the rule is allowed to resolve (covered in lesson 03). labels are routing identifiers that Alertmanager matches on. annotations are human-facing strings, optionally Go-templated, that ride with the alert.

Why a sysadmin cares

A rule without a label set reaches Alertmanager as an orphan. Alertmanager routes by label matchers; an alert that matches nothing falls through to the catch-all receiver, which is usually email or no-op. A rule without annotations reaches the on-call engineer as decoration: the alert says something is wrong but not what to do about it. Both failure shapes are common in rule estates that grew without a review discipline. The cost shows up the first time an alert fires at 03:00 and the on-call has no runbook link.

How it works

Prometheus evaluates rules on a wall-clock cadence. Each evaluation runs the expr, takes the result series, and advances the state machine for every series:

                  expr returns non-empty
   inactive  ---------------------------->  pending
      ^                                          |
      |                                          |  for: elapses,
      | expr returns empty                       |  expr still non-empty
      |                                          v
      +-------------------------------------  firing
                          expr returns empty,
                          keep_firing_for elapsed

The four observable states:

• inactive — the rule is loaded but the expr returned no series at this evaluation.
• pending — the expr returned at least one series, but the for: timer has not yet elapsed. Prometheus has not sent anything to Alertmanager yet.
• firing — the for: timer has elapsed and the expr still returns the series. Prometheus has sent the alert to Alertmanager; Alertmanager decides who to page.
• resolved — the expr no longer returns the series and keep_firing_for, if set, has elapsed. Prometheus has sent a resolution to Alertmanager; Alertmanager routes the resolution to its receivers.

The split between Prometheus and Alertmanager matters. Prometheus owns the lifecycle (when does the condition hold long enough). It also owns the labels and annotations. Alertmanager owns routing and notification. A rule that loads but never fires is a Prometheus problem. A rule that fires but never pages is an Alertmanager problem. Diagnostics differ.

How to configure it

A single rule, in production shape:

groups:
  - name: orders-api.slo
    interval: 30s
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        keep_firing_for: 30m
        labels:
          severity: critical
          team: checkout
          service: orders-api
          slo: availability
        annotations:
          summary: 'orders-api 5xx ratio above 5% for 5 minutes in {{ $labels.region }}'
          description: |
            The orders-api service in region {{ $labels.region }} has returned
            a 5xx ratio above 5% over the last 5 minutes. Current ratio:
            {{ $value | humanizePercentage }}.
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}'

The corresponding prometheus.yml:

global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

Each top-level key earns its place:

• interval: 30s overrides the global evaluation_interval for this group; useful when one group is expensive to evaluate.
• for: 5m is the anti-flap dwell time before firing.
• keep_firing_for: 30m keeps the alert firing for 30 minutes after the expr first returns empty, useful when an incident is long and the ratio oscillates across the threshold.
• labels carry routing information. severity and team are the two canonical ones; service and slo help with dashboard linking and SLO rollups.
• annotations carry human-facing context. summary is one short line; description is a paragraph; runbook_url and dashboard_url are stable links.

How to validate it

Three checks, in order. The first two are read-only; the third is read-only against a running Prometheus.

# 1. Static check: does the file parse, and is the expr syntactically valid?
promtool check rules /etc/prometheus/rules/orders-api.yml

Expected output:

Checking /etc/prometheus/rules/orders-api.yml
  SUCCESS: found 1 rules, 1 alerts

A non-zero exit with a parse error means the file does not load; fix the YAML and rerun before reloading Prometheus.

# 2. Live check: is the rule loaded by the running Prometheus?
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name == "OrdersApiHighErrorRate")'

Expected output during the for: dwell:

{
  "name": "OrdersApiHighErrorRate",
  "query": "sum by (service, region) (...)  > 0.05",
  "state": "pending",
  "evaluationTime": 0.012,
  "lastEvaluation": "2026-08-13T03:14:30.000Z",
  "keepFiringSince": null,
  "labels": { "severity": "critical", "team": "checkout" },
  "annotations": { "summary": "orders-api 5xx ratio above 5% ..." }
}

The state field is one of inactive, pending, firing. If the rule is missing from the response, the file failed to load; check rule_files glob and Prometheus logs.

# 3. The self-exposition: is Prometheus itself reporting an ALERTS series for this rule?
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS_FOR_STATE' \
  | jq '.data.result[] | select(.metric.alertname == "OrdersApiHighErrorRate")'

Expected output while pending:

{
  "metric": {
    "__name__": "ALERTS_FOR_STATE",
    "alertname": "OrdersApiHighErrorRate",
    "alertstate": "pending",
    "region": "eu-west-1",
    "service": "orders-api",
    "severity": "critical",
    "team": "checkout"
  },
  "value": [1723524870, "298.5"]
}

The value is the number of seconds the alert has been in the reported state. When the value crosses for:, the state becomes firing and Alertmanager receives the alert.

How it can fail

Six failure modes, each with an observable symptom:

1. Stays pending forever. Symptom: rule appears in /api/v1/rules with state: pending for hours; ALERTS_FOR_STATE{alertstate="pending"} keeps increasing. Cause: the expr returns series whose labels do not match an alert route, or for: is much larger than the window in which the expr actually fires. Confirm by computing the expr in Grafana Explore and watching it for the full for: duration.

2. Fires and resolves every minute. Symptom: Alertmanager inbox shows a fire-resolve-fire-resolve cycle. Cause: for: is too short relative to the dominant noise period on the expr. Tighten the expression, lengthen for:, or both.

3. Fires but no page arrives. Symptom: ALERTS{alertstate="firing"} is present, but Alertmanager logs no notification. Cause: the alert labels do not match any route in the route tree, or the catch-all is muted. Confirm by listing Alertmanager receivers and checking the alert payload in /api/v2/alerts.

4. runbook_url annotation link 404s. Symptom: on-call clicks the link and gets a 404. Cause: the runbook URL was hard-coded with a literal label value rather than a Go template, so the link points at the same fixed path for every series. Use {{ $labels.service }} and confirm the template renders by inspecting annotations in /api/v1/rules.

5. Rule loads with no errors but /api/v1/rules is empty for it. Symptom: promtool check rules says SUCCESS, but the rule does not appear in the running Prometheus. Cause: the file is not matched by the rule_files glob, or the file extension is not .yml/.yaml, or Prometheus was not reloaded after the glob changed. Reload Prometheus after fixing the glob.

6. promtool check rules exits non-zero after a Prometheus upgrade. Symptom: a rule that worked under 2.54 fails check rules under 2.55. Cause: a deprecated expression function was renamed or removed. The error message names the function and line; consult the release notes for the migration path.

How to troubleshoot it

In order:

1. Was the rule loaded? curl -s /api/v1/rules | jq '.data .groups[].rules[].name' | grep -F OrdersApiHighErrorRate. Empty result means the file did not load; check rule_files in prometheus.yml and the rule_loader log lines.
2. Does the expr return data? curl -G --data-urlencode 'query= <expr>' http://prometheus:9090/api/v1/query. Empty data.result means the metric is missing or the label selector is wrong.
3. What state is the alert in? curl /api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="OrdersApiHighErrorRate") | .state'. suppressed indicates an Alertmanager inhibition rule is hiding it.
4. What labels reached Alertmanager? Query the Alertmanager API: curl http://alertmanager:9093/api/v2/alerts | jq '.[] | select(.labels.alertname=="OrdersApiHighErrorRate")'. Confirm severity, team, service are present and correctly cased.
5. Was a notification attempted? Tail Alertmanager logs for the alert fingerprint. The nfdroid/notifier log lines show whether the webhook or pager integration actually fired.

If the rule loads, the expr returns series, and the state is firing, but no page arrives, the failure is on the Alertmanager side, not the Prometheus side. Do not edit the rule to fix it.

Security implications

Alert rule YAML is configuration, not data. A malicious rule cannot directly leak secrets, but a rule whose expr touches a high-cardinality label can cause a denial of service by exhausting the rule evaluator. Reviewers should reject rules whose expr includes unbounded labels (for example container_label without an allow-list) on first read.

The runbook_url and dashboard_url annotations are user-visible. Treat the URLs as content: confirm they point at trusted infrastructure, and avoid embedding credentials in URL paths because URLs end up in chat transcripts and ticketing systems.

Performance implications

Rules are evaluated in their entirety on every tick. A group with a wide expr (no label selectors) on a high-cardinality metric can dominate the rule evaluator. Common mitigations:

• Bound the expr with a label selector (service=~"orders|checkout| auth").
• Use sum by (...) to aggregate to a small number of series before the comparison.
• Set interval: on the group higher than the global evaluation_interval for expensive rules.
• Pre-compute the heavy aggregation as a recording rule and alert on the recording rule.

The limit: key on a group caps the number of series a single evaluation may produce. It is a safety net against an expr change that accidentally returns millions of series. Use it on groups that alert on raw counter or gauge metrics.

Production guidance

• Treat alert: as a stable identifier. Renaming a rule deletes its history in Alertmanager and breaks any dashboards that filter by alertname.
• Carry the minimum label set the Alertmanager routes need. Anything else goes in annotations.
• Set for: empirically. Start long (10m for service-internal, 5m for user-impact), tighten once you have observed the dominant noise source.
• Use keep_firing_for for incident-shaped rules that oscillate near the threshold; omit it for rules where resolution should propagate immediately (TLS cert expiry, backup freshness).
• Reload Prometheus with SIGHUP or POST /-/reload after any change to rule_files. Prometheus does not watch the filesystem.

Verification

• What is the difference between the pending and firing states, and which Prometheus metric exposes them?
• Why does a rule that parses cleanly still fail to produce alerts if the expr returns zero series?
• What does keep_firing_for change about a long-running incident, and when is it harmful to omit?
• Which two API endpoints on a running Prometheus let you confirm a rule is loaded and which state it is in?

Quiz