`for:` and Hysteresis

The first version of a rule says for: 30s. Two days later the on-call rota shows seven pages in 24 hours, every one of them the same alert firing on a 30-second scrape blip and resolving when the next scrape came in clean. The team tightens the rule with a better expression and bumps for: to 5m. Pages stop. The cost is that the same rule now takes five minutes longer to detect a real incident. The trade-off between flapping and detection latency is what for: exists to manage.

What it is

The for: clause on an alert rule is the dwell time the rule’s expr must continuously return a result series before the alert transitions from pending to firing. Its purpose is anti-flap: absorb short-lived breaches of the threshold so the alert does not toggle every evaluation.

keep_firing_for (added in Prometheus 2.42) is the symmetric post-fire dwell time. Once the rule has fired, the alert stays firing for at least keep_firing_for even if the expr returns empty. Its purpose is anti-resolve: prevent the alert from chattering into resolution during a long incident that oscillates near the threshold.

Together, for: and keep_firing_for describe a hysteresis band: the alert enters firing after for: of continuous breach and exits firing after keep_firing_for of continuous recovery.

Why a sysadmin cares

A rule with no for: (or for: 0s, the default) fires on every evaluation where the threshold is crossed. On a 30-second scrape interval, a 90-second blip produces three notifications. The Alertmanager group_wait and group_interval smooth some of this, but the underlying signal is noisy and the on-call rota pays the cost.

A rule with for: set too long misses real incidents. A rule with keep_firing_for set too long makes the alert look stuck when it has actually recovered. Both tuning choices are operational judgements, and both are testable.

How it works

The state machine is the same one introduced in lesson 01, with two time-bound transitions:

   evaluation_interval ticks
   |
   v
   expr evaluated
   |
   +--- empty ---> if pending  -> inactive (timer reset)
   |               if firing   -> if keep_firing_for elapsed -> resolved
   |                                  else stay firing
   +--- non-empty
           |
           +--- no prior alert for this series
           |        -> create pending alert
           |           (start for: timer)
           |
           +--- prior alert in pending
           |        -> tick for: timer; if elapsed and expr
           |           still non-empty -> transition to firing
           |
           +--- prior alert in firing
                    -> reset keep_firing_for timer; stay firing

The two time-bound transitions are:

• pending to firing, gated by for:.
• firing to resolved, gated by keep_firing_for (or, if absent, by the expr returning empty for one evaluation).

A worked example. A rule with for: 5m and an evaluation interval of 30s:

  t=00:00  expr returns non-empty  alert created, pending
  t=00:30  expr returns non-empty  pending, timer at 30s
  t=01:00  expr returns non-empty  pending, timer at 60s
  t=01:30  expr returns empty      pending -> inactive, timer reset
  t=02:00  expr returns non-empty  pending recreated, timer at 0
  t=02:30  expr returns non-empty  pending, timer at 30s
  ...
  t=07:00  timer hits 5m, expr     firing, Alertmanager notified
          still non-empty

The key behaviours:

• The timer resets whenever the expr returns empty.
• A single empty evaluation is enough to reset the timer.
• evaluation_interval does not change the for: budget; it only changes how often the timer is checked. With a 30s interval and for: 5m, the rule still fires after 5 minutes of continuous breach, not 5 minutes plus one interval.

How to configure it

A worked pair. First, the rule for a user-impact error budget:

groups:
  - name: orders-api.slo
    interval: 30s
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        keep_firing_for: 30m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: 'orders-api 5xx ratio above 5% for 5 minutes in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

      - alert: TlsCertExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
          team: platform
        annotations:
          summary: 'TLS cert for {{ $labels.instance }} expires in less than 14 days'
          runbook_url: 'https://runbooks.example.com/platform/rotate-tls'

Two rules, two anti-flap strategies:

• The error-rate alert pages on a sustained breach (for: 5m) and stays firing through a long incident (keep_firing_for: 30m). The for: is longer than the typical GC pause or deploy blip. The keep_firing_for is longer than the typical oscillation period during a regional incident.
• The TLS alert uses for: 1h with no keep_firing_for. The cert is a discrete signal; once it has been below 14 days for an hour, the alert should fire and stay firing until the cert is actually rotated.

Three knobs and the trade-offs:

How to validate it

Three checks. The first two are read-only against a running Prometheus; the third is a unit test.

# 1. Inspect the live state and the dwell counter.
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS_FOR_STATE' \
  | jq '.data.result[] | select(.metric.alertname == "OrdersApiHighErrorRate")
        | {state: .metric.alertstate, seconds: .value[1]}'

Expected output during the dwell:

{
  "state": "pending",
  "seconds": "298.5"
}

The seconds value should be increasing on every evaluation. If it jumps back to a small number, the expr returned empty in the interim and the timer reset.

# 2. Inspect the alert state itself.
curl -s http://prometheus:9090/api/v1/alerts \
  | jq '.data.alerts[] | select(.labels.alertname == "OrdersApiHighErrorRate")
        | {state, activeAt, keepFiringSince}'

Expected output once firing:

{
  "state": "firing",
  "activeAt": "2026-08-13T03:09:30.000Z",
  "keepFiringSince": "2026-08-13T03:14:30.000Z"
}

activeAt is when the alert entered pending. keepFiringSince is when it transitioned to firing. The gap is the dwell plus evaluation jitter.

# 3. Unit-test the rule with promtool. The test file (covered in
# detail in lesson 06) replays a synthetic series and asserts
# that the rule fires only after the for: dwell.
promtool test rules test_orders_api.yml

How it can fail

Six failure modes:

1. Flap at the scrape interval. Symptom: the alert fires and resolves every 60 seconds. Cause: for: is shorter than the dominant noise period, or the expr is intrinsically noisy (e.g. a count over a 1-minute window). Lengthen for: or rewrite the expr over a longer window.

2. Pending forever, never firing. Symptom: the alert has been in pending for hours. Cause: the expr returns series only briefly, so the for: timer resets before it elapses. Confirm by watching the expr in Grafana Explore.

3. Real incident delayed by for:. Symptom: the on-call rota knew something was wrong from dashboards, but the alert did not fire until 10 minutes in. Cause: for: is too long for the user-impact threshold. Tighten for: and add a parallel shorter-for: rule at lower severity.

4. Resolves during a long incident. Symptom: a 90-minute outage produces three resolve-and-re-fire cycles. Cause: the expr oscillates near the threshold during partial recovery. Add keep_firing_for at a value larger than the typical oscillation period.

5. Alert appears stuck after recovery. Symptom: the rule keeps firing long after the underlying metric is healthy. Cause: keep_firing_for is too long. Either shorten it or remove it for rules where immediate resolution is desirable.

6. for: larger than the scrape interval, but the rule never evaluates long enough to fire. Symptom: the alert remains in pending. Cause: evaluation_interval is shorter than the scrape interval, and the metric is missing for an extended period. Use absent() or add a synthetic zero to the series so the rule can observe the gap.

How to troubleshoot it

In order:

1. Inspect the current state. /api/v1/alerts and /api/v1/query?query=ALERTS_FOR_STATE. The state and the dwell counter tell you whether the rule is waiting, firing, or stuck.
2. Inspect the expr over the same window. Compute the expr in Grafana Explore for the past hour. If the expr returned empty for any sub-window, the for: timer reset.
3. Inspect the scrape health. up{job="..."} shows whether the source was being scraped. A scrape gap shows up as a hole in the metric and a reset in the rule.
4. Inspect the eval interval. Confirm evaluation_interval on the rule group. A 5-minute for: with a 1-minute interval is fine; a 5-minute for: with a 30-second interval does not fire faster.
5. Re-tune empirically. Change for: and keep_firing_for in a non-production replica. Watch the rule against synthetic load. Promote once the trade-off looks right.

Security implications

for: and keep_firing_for are pure timing parameters with no security implications on their own. The expr they gate, however, runs every evaluation. A rule whose expr is a heavy aggregation over a high-cardinality metric can be used to amplify a small input into an expensive rule evaluation; for: does not mitigate that. Bound the expr with label selectors before lengthening for:.

Performance implications

A short evaluation_interval on an expensive group multiplies CPU cost. The cheap path is to lengthen evaluation_interval on the group rather than shortening for:; the two are independent knobs and serve different purposes. A common pattern:

groups:
  - name: expensive-rollups
    interval: 5m
    rules:
      - alert: SomethingOverThreshold
        expr: <expensive>
        for: 10m
        keep_firing_for: 30m

The expensive expr runs every 5 minutes; the for: provides 10 minutes of dwell; keep_firing_for keeps the alert sticky during a long incident. None of the three knobs fight each other.

Production guidance

• Start with for: 5m for user-impact alerts, for: 15m for service-internal alerts. Tighten only after observing the noise pattern.
• Set keep_firing_for on rules where the expr can oscillate near the threshold during a long incident (error rates, saturation, capacity). Omit it where resolution should propagate immediately (TLS expiry, backup freshness).
• Match for: to the dominant noise period, not the scrape interval. Scrape blips are absorbed by for:; longer noise needs a longer for:.
• Match keep_firing_for to the dominant oscillation period during an incident. If you do not know, leave it unset and add it when the alert starts to flap.

Verification

• What transition does for: gate, and what transition does keep_firing_for gate?
• What happens to the for: timer when the expr returns empty for one evaluation while the alert is in pending?
• How does evaluation_interval interact with for:? Does a shorter evaluation_interval fire the alert sooner?
• When is keep_firing_for harmful rather than helpful?

Quiz