Alert Time-to-Fire

A team sets every alert with for: 5m. The reasoning is consistent: five minutes is enough to absorb a scrape gap without flapping. The reasoning is also wrong. The checkout error-rate alert fires after five minutes; the SLO burn-rate alert fires after five minutes; the disk-capacity alert fires after five minutes; the certificate-expiry alert fires after five minutes. The on-call rota is paged for a real customer-visible checkout failure five minutes after the failure begins — and the certificate that expires in seven days pages five minutes before it expires, after the customer-visible failure already happened.

The mistake is treating time-to-fire as a single value. The right discipline is a tiered time-to-fire budget that maps each alert’s operational impact to a distinct dwell. A customer-visible failure should fire in seconds; a seven-day certificate expiry can wait hours; a slow disk fill can wait fifteen minutes. The trade-off is between flapping (too short) and missed detection (too long).

What it is

Time-to-fire is the wall-clock duration between the moment the alert condition becomes true in the live system and the moment the alert fires in Alertmanager. The duration has three components:

  +---------------------------------------------+
  |  1. Scrape interval                         |
  |    The Prometheus scrape interval. A        |
  |    15-second scrape means a 0-15 second     |
  |    delay before the metric is visible to    |
  |    the rule evaluator.                      |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  2. Rule evaluation interval                |
  |    The rule group's interval. A 30-second   |
  |    interval means the rule evaluates every  |
  |    30 seconds; the alert condition is       |
  |    observed 0-30 seconds after the scrape.  |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  3. for: dwell                              |
  |    The rule's for: clause. The alert must   |
  |    be true for this duration before it      |
  |    fires. A 5-minute dwell means the alert  |
  |    is pending for 5 minutes before it       |
  |    transitions to firing.                   |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  4. Alertmanager group_wait                 |
  |    The time Alertmanager waits before       |
  |    sending the notification. A 10-second   |
  |    group_wait means up to 10 seconds from   |
  |    firing to notification delivery.         |
  +---------------------------------------------+

The total time-to-fire is the sum:

  time_to_fire = scrape_interval
               + rule_interval
               + for_dwell
               + group_wait

For a typical rule (15s scrape, 30s interval, 5m dwell, 10s group_wait), the time-to-fire is 5 minutes 55 seconds at worst, 5 minutes 10 seconds at best. The dwell dominates.

The right approach is to choose the dwell based on the alert’s tier. The tier maps the operational impact to a budget.

The most common shape is a four-tier time-to-fire budget:

A symptom alert that pages within 1-2 minutes catches the real customer-visible failure before customers tweet. An SLO burn-rate alert that pages within 1-5 minutes catches the burn before the error budget is exhausted. A capacity alert that pages within 15-30 minutes gives the on-call enough time to act before the disk is full. A canary alert that fires on a fixed schedule verifies the chain. A lead-time alert pages days before the deadline so the team can plan.

Why a sysadmin cares

The time-to-fire budget is the trade-off between flapping and missed detection.

• Too short (e.g., for: 0m). The alert fires on every scrape gap, every transient spike, every brief network blip. The on-call rota is paged for non-events. Alert fatigue sets in; the alert is muted; the next real page is triaged at the bottom of the queue.
• Too long (e.g., for: 30m). The alert does not fire until the condition has been true for thirty minutes. The customer-visible failure goes undetected for thirty minutes. The on-call rota wakes to a full-blown incident instead of an early signal.

The right dwell is the value that catches the real condition without flapping on the transient. The discipline is to map each alert’s operational impact to a tier and to use the tier’s budget as the dwell.

How it works

The rule’s for: clause is a dwell: the alert must be true for the configured duration before it fires. The dwell counts in evaluation ticks. A rule with for: 5m and a 30s evaluation interval fires after the alert condition has been true for 10 consecutive evaluation ticks.

  Tick  Condition  State
  ----  ---------  -----
  0     false      inactive
  1     true       pending
  2     true       pending
  3     true       pending
  4     true       pending
  5     true       pending
  6     true       pending
  7     true       pending
  8     true       pending
  9     true       pending
  10    true       firing   <-- dwell elapsed
  11    true       firing

The dwell is the number of consecutive ticks the condition must be true. If the condition becomes false on tick 5, the pending state resets; the dwell starts over on the next tick where the condition is true.

The tiered time-to-fire budget is implemented as a per-tier for: value:

The tier’s for: value is set in the rule’s YAML. A unit test fixture asserts the alert fires within the tier’s budget against a synthetic series. A performance budget test asserts the time-to-fire in production does not exceed the tier’s budget.

How to configure it

A worked example for a checkout service with four tiers:

The rule file:

# observability/prometheus/rules/checkout.yml
groups:
  - name: checkout-symptom
    interval: 15s
    rules:
      # Tier 1: Symptom alert. Fires within 2 minutes of the
      # customer-visible failure.
      - alert: CheckoutHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api",
                                     status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 2m
        labels:
          severity: critical
          team: checkout
          tier: symptom
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

  - name: checkout-capacity
    interval: 1m
    rules:
      # Tier 3: Capacity alert. Fires within 30 minutes of
      # disk fill crossing 85%.
      - alert: CheckoutDiskFilling
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600)
            < 0
          and
          node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.15
        for: 30m
        labels:
          severity: warning
          team: checkout
          tier: capacity
        annotations:
          summary: 'orders-api disk predicted to fill in 4h'
          runbook_url: 'https://runbooks.example.com/checkout/disk-fill'

  - name: checkout-certificate
    interval: 1h
    rules:
      # Tier 5: Lead-time alert. Pages 14 days before
      # certificate expiry.
      - alert: CheckoutCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry_seconds - time() < 14 * 86400
        for: 1h
        labels:
          severity: warning
          team: checkout
          tier: lead-time
        annotations:
          summary: 'orders-api TLS cert expires in less than 14 days'
          runbook_url: 'https://runbooks.example.com/checkout/cert-expiry'

Three rules. Each carries a tier: label that identifies its time-to-fire budget. The Symptom alert has for: 2m; the Capacity alert has for: 30m; the Lead-time alert has for: 1h (the dwell is short because the lead time is the time-to-cert-expiry, not the dwell).

The unit test fixture that asserts the tier’s budget:

# observability/prometheus/rules/test/checkout_test.yml
rule_files:
  - ../checkout.yml

evaluation_interval: 15s

tests:
  # Tier 1: Symptom alert fires within 2 minutes.
  - interval: 15s
    name: symptom alert fires within 2 minute budget
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 10 10 10 10 10 10 10 10 10 10 10 10'
    alert_rule_test:
      - eval_time: 1m
        alertname: CheckoutHighErrorRate
        exp_alerts: []  # pending, not firing, within the dwell
      - eval_time: 3m
        alertname: CheckoutHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: checkout
              tier: symptom
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5% in eu-west-1'

  # Tier 1: Symptom alert does not fire on transient spike.
  - interval: 15s
    name: symptom alert does not fire on a 30 second spike
    input_series:
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="200"}'
        values: '0 100 100 100 100 100 100 100 100 100 100'
      - series: 'http_requests_total{service="orders-api",region="eu-west-1",status="500"}'
        values: '0 10 0 0 0 0 0 0 0 0 0'  # spike at t=15s, then clear
    alert_rule_test:
      - eval_time: 2m
        alertname: CheckoutHighErrorRate
        exp_alerts: []  # pending state did not accumulate; the
                       # spike cleared before the dwell elapsed

The first test asserts the symptom alert fires within the 2-minute budget. The second test asserts the symptom alert does not fire on a transient spike that clears before the dwell elapses.

How to validate it

Three checks confirm the time-to-fire discipline is in place.

1. Every rule carries a tier: label.

curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[]
        | select(.type=="alerting")
        | {alert: .name, tier: .labels.tier, for: .for}'

Expected output: every rule has a tier label. A rule without a tier label indicates the discipline is not applied.

2. The unit test asserts the tier’s budget.

promtool test rules \
  observability/prometheus/rules/test/checkout_test.yml

Expected output, exit 0:

SUCCESS

A failing fixture means the tier’s budget is not met.

3. The time-to-fire in production matches the tier.

# Trigger a synthetic condition; measure the time-to-fire.
START=$(date +%s)
curl -s -X POST http://synthetic-app:9101/admin/fire
# Wait for the alert to fire (poll Alertmanager)
while true; do
  STATE=$(amtool alert query alertname=CheckoutHighErrorRate \
    | awk '/firing/ {print $3; exit}')
  if [ "$STATE" = "firing" ]; then
    END=$(date +%s)
    break
  fi
  sleep 5
done
echo "Time-to-fire: $((END - START)) seconds"

Expected output, for a symptom alert with a 2-minute budget:

Time-to-fire: 135 seconds

A time-to-fire significantly longer than the tier’s budget indicates the dwell is wrong or the interval is too long.

How it can fail

Six failure modes appear repeatedly when teams adopt a tiered time-to-fire discipline.

1. Every alert has the same for: value. Symptom: a customer-visible failure and a seven-day certificate expiry both page after the same dwell. Cause: the team picked a single value for consistency. Fix: assign each rule to a tier and use the tier’s budget.
2. The dwell is too short for a transient-prone metric. Symptom: the alert flaps on every brief spike. Cause: the dwell is shorter than the metric’s normal variance. Fix: measure the metric’s variance; set the dwell to at least 3 times the 95th percentile of transient duration.
3. The dwell is too long for a customer-visible failure. Symptom: the on-call rota wakes to a full-blown incident instead of an early signal. Cause: the tier’s budget is not matched to the operational impact. Fix: use the symptom tier’s 1-2 minute budget for customer-visible failures.
4. The interval is too long for the tier’s budget. Symptom: a 2m dwell with a 1m interval means the alert fires after 2 ticks at worst, 1 tick at best; the dwell is effectively 1-2 minutes, not 2 minutes. Cause: the interval is not tuned to the dwell. Fix: set the interval so the number of ticks within the dwell is appropriate (e.g., 8 ticks for a 2m dwell).
5. The unit test does not assert the dwell arithmetic. Symptom: a rule with for: 2m and interval: 5m ships to production; the alert fires after one tick at worst, or never at all if the condition is brief. Cause: the fixture does not span the dwell. Fix: extend the input_series to cover the dwell window.
6. The Alertmanager group_wait adds latency the team did not budget for. Symptom: the rule fires at the tick but the notification arrives seconds later. Cause: group_wait is set to 30s or longer. Fix: set group_wait to 10s or less for symptom alerts; account for it in the tier’s budget.

How to troubleshoot it

In order:

1. What tier is the rule in? curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name=="X") | .labels.tier'. A missing tier label indicates the rule is not in the discipline.
2. What is the rule’s for: and interval:? curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name=="X") | \{for: .for, interval: .interval\}'. A for: that does not match the tier’s budget indicates the rule is misconfigured.
3. Does the unit test pass? promtool test rules <fixture>. A failing fixture means the dwell arithmetic is wrong.
4. What is the actual time-to-fire in production? Run the synthetic trigger script and measure the time from trigger to firing. A value significantly longer than the tier’s budget indicates a misconfiguration.
5. What is the Alertmanager group_wait? grep group_wait alertmanager.yml. A group_wait longer than 10s for a symptom alert adds latency the tier’s budget did not account for.

Security implications

• The time-to-fire budget is not a security control. An alert that fires in 1 minute is not “more secure” than one that fires in 5 minutes. The budget is an operational trade-off.
• The tier: label is metadata. A label that identifies an alert as a symptom alert does not leak sensitive information. The label can be exposed in Alertmanager payloads without redaction.
• A flap-suppression configuration that masks real alerts is a security risk. A flap-suppression window that is too long may suppress a real alert that fires within the budget. Tune the suppression window separately from the dwell.

Performance implications

• The interval drives the rule evaluation cost. A rule with interval: 15s evaluates 4 times per minute. A rule with interval: 1m evaluates 1 time per minute. Symptom alerts at 15s intervals are the most expensive; capacity alerts at 1m intervals are cheap. The total cost is dominated by the symptom alerts.
• The dwell does not affect the evaluation cost. The dwell is a state machine on top of the evaluation; the rule evaluates the same number of times regardless of the dwell. A 2m dwell and a 30m dwell both evaluate at the configured interval.

Production guidance

• Adopt a tiered time-to-fire budget. A team that uses a single value for every alert catches neither fast failures nor slow fills well. A team that uses a tiered budget maps each alert’s operational impact to a distinct dwell.
• Carry a tier: label on every rule. The label identifies the alert’s tier and the budget it should meet. A rule without a tier: label indicates the discipline is not applied.
• Tune the interval to the tier’s budget. A symptom alert with a 2m budget should evaluate every 15s (8 ticks). A capacity alert with a 30m budget can evaluate every 1m (30 ticks). The interval is part of the tier’s budget.
• Unit-test the dwell arithmetic. The fixture must span the dwell window; the assertion must check both the pending and firing states; the assertion must check the transient scenario (a brief spike that clears before the dwell elapses).
• Account for Alertmanager group_wait. The notification delivery adds latency to the tier’s budget. Symptom alerts should have a 10s group_wait; capacity alerts can tolerate a 30s group_wait.

Verification

You should now be able to answer:

• What are the four components of the time-to-fire duration, and which component dominates?
• What is the tiered time-to-fire budget, and what is the typical dwell for each tier?
• Why must the rule’s interval: be tuned to the tier’s budget?
• What is the difference between the symptom tier and the capacity tier in terms of dwell and operational intent?
• Why must the unit test fixture span the dwell window and assert both the pending and firing states?

Quiz