`for:` Too Short

A PagerDuty alert appears at 14:32:10. By 14:32:55 it has resolved itself. Total time in the firing state: 45 seconds. Total notification cost: a page, a wake-up of the on-call engineer, a Slack thread, an open laptop, a closed laptop. The condition was a single 15-second scrape where CPU crossed 85% during a routine garbage collection pause. The rule read:

- alert: ApiHighCPU
  expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[1m])) > 0.80
  for: 30s

The for: 30s clause is not the smoothing window on CPU. The 1-minute rate() window is the smoothing. for: 30s is the dwell time in pending before the alert is allowed to advance to firing. CPU paused past 80%, the rate over the last minute crossed 80%, the pending entry for that label set was 30 seconds old at the next evaluation, the alert advanced to firing. The pause ended, the rate dropped, the alert resolved. The page was over.

What for: is

for: is the minimum dwell time in the pending state that a series must satisfy before the rule is allowed to advance to firing. The clause is applied per series, per evaluation, on the same labels that the expression selects.

Three things that for: is not:

• It is not a smoothing window on metric values. The smoothing is inside the PromQL function (rate, increase, avg_over_time).
• It is not a rate limit on notifications. That lives in Alertmanager (route grouping, repeat_interval).
• It is not a minimum duration of the underlying condition. The condition can persist for a single evaluation beyond for:, fire, and resolve on the evaluation after that.

A rule with for: 5m against an expression that selects a series that crosses threshold for 4m59s and resolves will never fire. The pending entry expires; the alert does not advance. A rule with for: 0s against the same condition fires for the single evaluation that satisfies the expression and resolves on the next.

Why a sysadmin cares

A for: that is too short produces five pain shapes:

• One-shot pages. A 45-second page is enough to lose context, walk to a laptop, open Grafana, and decide the incident is over by the time the engineer gets there. The on-call engineer learns the rule is a pager of last resort.
• Webhook flood. Alertmanager groups repeated short alerts on the same fingerprint, but the grouping log still records each transition. The webhook receivers fill with firing → resolved → firing → resolved events.
• Notification system fatigue. Pushover, Slack, PagerDuty all treat rapid-fire short events as noise. Some receivers rate-limit. The real page gets stuck behind the noise.
• Runbook impossible. A 45-second incident does not invite a runbook walk. The on-call engineer improvises.
• Trace correlation lost. If the alert resolves before a trace is sampled, the investigation has nothing to point at. The team wonders what happened.

A for: that is too long produces a different pain: the incident has already manifested by the time the alert fires. A for: 30m rule against a saturation event that produces customer pain at minute 5 is, in operational terms, silent on the customer-visible time. Lesson 03 is about the too-short failure shape; the too-long shape is part of Lesson 05’s investigation discussion.

How it works

The alert state machine is per series, not per expression.

[inactive] --(expression returns a series for these labels)
    |
    v
[pending]  --(series returned for `for:` duration)
    |
    v
[firing]   --(sent to Alertmanager)
   |
   +--- (expression no longer returns the series)
   |
   v
[inactive] (next evaluation where expression returns vector
            without these labels, the pending list drops)

Two timing properties matter:

• The for: clock is wall-clock against the evaluation interval. A for: 1m rule evaluated every 30 seconds fires after two consecutive evaluations that satisfy the expression, not after a fixed wall-clock minute.
• keep_firing_for (added in Prometheus 2.42) is the dual of for:. It holds the alert in firing for keep_firing_for after the underlying condition has cleared, to prevent flapping. for: controls entry to firing; keep_firing_for controls exit.

The most common shape

The too-short pattern is well known and well-typed. Most production for: failures look like one of these:

1. A for: value copied from a community rule. The sample rule in the Prometheus docs uses for: 10m for up == 0. The sample rule in a community repo uses for: 30s for the same expression. Community values sometimes make sense for the rule the community writes and never make sense for the rule the team has.

2. A for: value chosen to match the rate window. A team writes rate(...[5m]) and sets for: 5m, believing the two numbers must agree. They do not. The for: is about how long the smoothed rate must remain above threshold; setting it equal to the rate window collapses the two and creates a one-breach alert.

3. A for: value at the autoscaling period. A horizontal pod autoscaler adds pods every 60 seconds; the alert for: 30s fires during the scale-out, resolves after the new pods are warm. The team thought they were watching the application, but they were watching the autoscaler.

4. A for: value shorter than the scrape interval plus jitter. A scrape interval of 15s with a for: 10s fires on a single 5xx sample that lands on the wrong boundary. This is the dominant shape in services whose scrape intervals are short.

The lesson’s pattern: a too-short for: always sits at the same scale as some other clock in the system. Identify that clock, and the fix is to lift for: past it.

Under the hood

In Prometheus 2.55.x, the alerting engine tracks pending and firing entries per label set per alert. The WAL records the notification transitions and replays them on restart. The for: value is part of the rule file, parsed at load time; changing it is a config reload, not a runtime parameter.

A rule that uses a 5-minute rate() and a 30-second for: has two separate timing axes:

• The rate window (5 minutes) smooths the underlying metric. A 30-second saturation event shows up as a 30/300 contribution to the smoothed rate.
• The for: window (30 seconds) gates the alert advancement, not the rate.

The two interact. A 30-second saturation produces a single-scrape crossing of the smoothed rate (because 30s out of 300s is 10%, which is usually enough to push the smoothed rate past most thresholds). Combined with for: 30s, the alert advances.

For an alert to be silent on a 30-second blip, either the smoothed rate must not cross (the rate window must be larger than the blip, weighted by the blip’s magnitude) or the for: must be longer than the blip’s lifetime. These are independent levers.

How to configure it

The configuration change is in the rule, and only in the rule. The shape of a healthy for: block:

groups:
  - name: api.rules
    interval: 30s
    rules:
      - alert: ApiCPUHighSustained
        expr: |
          avg by (instance) (
            rate(
              node_cpu_seconds_total{
                mode!="idle",
                job="api-svc",
                environment="production",
              }[5m]
            )
          )
          > 0.85
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: page
          team: api
        annotations:
          summary: 'API CPU above 85% sustained for 10m on {{ $labels.instance }}'
          runbook: 'https://runbooks.example.com/api/cpu'
          for_basis: 'set above 1 autoscaling cycle + 1 rate window'

Three things to read into that:

• rate(...[5m]) smooths over 5 minutes. Bursts shorter than ~30 seconds do not push the smoothed rate above 0.85.
• for: 10m is longer than the autoscaling period (60s plus warm-up), longer than the rate window (5m), and long enough that one scrape-off blip will not advance the alert.
• keep_firing_for: 5m (Prometheus 2.42+) holds the alert in firing for 5 minutes after the underlying condition has cleared. Without it, a rule that flaps will produce a firing → inactive → firing sequence every time the metric oscillates around the threshold.

How to validate it

Validate by reading the alert history. The state transition log lives in Prometheus and in Alertmanager.

curl -s 'http://prometheus:9090/api/v1/rules' \
  | jq '.data.groups[].rules[] | select(.name=="ApiCPUHighSustained") | .health'

Sample output during normal operation:

"ok"

With the alert active and for: holding:

curl -s 'http://prometheus:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="ApiCPUHighSustained") | {state, activeAt, value}'

Sample output while the alert is held by for::

{
  "state": "pending",
  "activeAt": "2026-08-13T14:31:50.234Z",
  "value": "0.912"
}

The state: pending confirms the alert has been tracking for less than the for: duration. Watch the state transition under a controlled test load (Lesson 05 in this module treats the procedure). The validation rule is: under a 30-second test burst that pushes CPU past 0.85, the alert must NOT advance to firing. The fix is to lift for: until the test passes.

promtool check rules /etc/prometheus/rules/api.yml

SUCCESS: rule files validated; 6 rules found, 0 errors

How it can fail

Six specific shapes, each with the symptom that distinguishes it from other for: failures:

1. Shorter than a single scrape interval. With scrape_interval: 15s and for: 10s, a single 5xx sample advances the alert. Symptom: the alert opens and resolves inside two consecutive evaluations.

2. Shorter than one rate window. A rule with rate(...[5m]) and for: 1m fires on a single 1-minute spike inside the rate window. Symptom: the alert opens when the spike hits, resolves when it dissipates, and the total duration equals the spike lifetime.

3. Shorter than the autoscaling warm-up. A horizontal pod autoscaler adds pods every 60s with 90s warm-up; a for: 60s rule fires during scale-out, resolves once the new pods are warm. Symptom: the alert correlates one-for-one with autoscaler events.

4. Shorter than the GC pause. A Java service with a 60s GC pause; for: 30s fires during the pause, resolves after. Symptom: the alert correlates with GC logs.

5. No keep_firing_for and the metric oscillates. The alert flaps between firing and inactive every few evaluations. Symptom: five PagerDuty events in three minutes; Alertmanager groups them but the webhook log shows the churn.

6. for: 0s and the rule depends on for: for debouncing. A team uses for: 0s because they wanted “immediate” alerts and forgot that for: is a debouncer. Symptom: an alert whose entire purpose is to debounce is firing on every scrape.

How to troubleshoot it

1. Open the rule. Read the for: value. Read the rate window in the expr. Read the scrape interval of the target. Note the autoscaling period of the workload.
2. Read the alert state history over the last 7 days. Identify the duration distribution of firing → inactive transitions. The shape of that distribution tells you what for: is currently calibrated against: one-scrape alerts (30s) suggest for: shorter than one rate window; aligned-with-deployment alerts (60-90s) suggest for: shorter than autoscaling.
3. Pick the boundary the team wants to defend. A reasonable default is “above 2× the longest system clock you want to debounce on”. A 5-minute rate plus 1-minute scrape plus 60s autoscaling suggests for: 5m as a starting point.
4. Apply the change. Run promtool check rules. Reload Prometheus. Run a controlled burst and verify the alert stays in pending for the duration of the burst.

Security implications

A for: that is too short on a security metric produces detection latency under one second. A for: that is too short on a brute-force detection rule is a feature, not a defect — the attacker is operating on the same timescale.

For authentication and authorisation metrics, calibrate for: to the attacker’s expected dwell time, not to the engineer’s preferred blip tolerance. The same for: is correct for a CPU rule and wrong for a failed-login rule.

Performance implications

Rule evaluation cost is unaffected by for:. The notification cost is: a for: that is too short inflates the firing/inactive transition log; Alertmanager 0.28.x’s notification grouping handles grouping on fingerprint but does not throttle the underlying log entries. A high transition rate also pushes more entries into the WAL, slowing recovery on restart.

Production guidance

• Calibrate for: against the slowest blip the team wants to debounce. Two times the rate window is a starting point, not a rule.
• Pair for: with keep_firing_for (Prometheus 2.42+) on rules whose underlying metric oscillates.
• Treat any for: value below 1 minute as suspect. Most service clocks (autoscaling, GC, batch) operate on scales of minutes or longer; for: 30s rarely makes sense.
• Recompute for: after a service changes its autoscale profile or introduces a new batch job.

Verification

You should now be able to answer:

• What does for: actually gate, and what does it not gate?
• What is the difference between for: and the rate window in a rate() expression?
• What symptom tells you the alert is firing inside a single rate window?
• Why is keep_firing_for part of the answer to a flapping rule, even though the lesson focused on for:?

Quiz