Rate Window Selection

Three dashboards land at the same weekly review. The first plots rate(http_requests_total[30s]). The panel is a sawtooth; every scrape is its own data point; the spike at 02:55 is invisible inside the noise. The second plots rate(http_requests_total[1m]). The panel is smoother but still shows the spike at 02:55, and the alert rule that pages on this metric fired twice during the spike and resolved between firings. The third plots rate(http_requests_total[5m]). The panel is smooth, the spike is barely visible, and the alert rule did not fire because the 5-minute average did not cross the threshold.

All three teams are using rate(). All three have configured it correctly according to its own definition. All three are getting a different operational answer. The window choice is the lever that determines the answer.

This lesson is about the rate window: how to choose it, how it interacts with the scrape interval and the SLO window, and how to configure it as code.

What the rate window is

The rate window is the duration in square brackets after a counter in a rate() or increase() expression. It controls two things:

1. How many samples the evaluation uses. A window of [1 m] on a 15 s scrape interval contains 4 samples; a window of [5 m] contains 20 samples.
2. What the evaluation smooths over. A spike that lasts 30 s is visible in a [1 m] window and barely visible in a [5 m] window.

The window is not the same as the Grafana min step. The min step is the resolution of the graph; the rate window is the stability of the evaluation. Both matter; they are configured independently.

Why a sysadmin cares

The window choice is the difference between a dashboard that pages on real incidents and a dashboard that pages on nothing, or never pages. It is also the difference between an SLO measurement that catches a breach and one that averages it away.

Three operational questions, three correct windows:

• “Is the service healthy right now?” A short window ([1 m]) catches the change in seconds. The cost is noise from single-scrape variance and from counter resets during deploys.
• “Are we burning the SLO error budget?” A medium window ([5 m]) is the standard for SLO error budgets. The window matches the typical SLO measurement step and avoids the noise that produces false alerts.
• “What is the long-term traffic trend?” A long window ([15 m] or [30 m]) smooths over deploys and short spikes. The panel shows the trend, not the event.

The wrong window produces a dashboard that answers the wrong question. The team that built it thought they were answering the right one.

How window selection works

The window is governed by four interacting constraints:

1. Scrape interval. The window must contain at least two samples. The 2x to 4x rule says: window should be 2x to 4x the scrape interval. With a 15 s scrape, [30 s] to [1 m] is the minimum stable window.
2. Counter reset cadence. If the counter resets on every deploy, the window should span at least one full deploy cycle. A typical Kubernetes rolling restart takes 2 to 5 minutes per pod; a cluster-wide rollout takes 10 to 30 minutes. The window should match.
3. SLO measurement step. An SLO measured at 5-minute resolution wants a 5-minute window. An SLO measured at 1-hour resolution wants a longer window. The window determines the smoothing of the SLO input.
4. Operational latency. The window determines how long it takes a real change to become visible. A 1-minute window shows the change within a minute. A 5-minute window shows it within 5 minutes.

The four constraints are in tension. A shorter window is more responsive but less stable. A longer window is more stable but slower to detect. The choice is the trade-off.

  window        samples   resets absorbed   ops latency   SLO use
  --------      -------   ---------------   -----------   -------
  [15s]         1 to 2    none              ~15 s         do not use
  [30s]         2         none              ~30 s         do not use
  [1m]          4         single            ~1 min        short alert
  [2m]          8         single            ~2 min        ops panel
  [5m]          20        rolling restart   ~5 min        SLO standard
  [15m]         60        full rollout      ~15 min       trend panel
  [1h]          240       full rollout      ~1 hour       capacity

The table is industry convention, not PromQL law. The exact boundaries depend on the deploy cadence and the SLO measurement step.

Under the hood

How to configure it

The window is configured in the PromQL expression. Recording rules are the production-grade way to share the right window across dashboards and alerts.

Recording rules for ops dashboards. A short window for ops panels, evaluated every 30 seconds:

# /etc/prometheus/rules/http-rate.yaml
groups:
  - name: http-rate
    interval: 30s
    rules:
      - record: job:http_requests:rate1m
        expr: |
          sum by (job, status) (
            rate(http_requests_total[1m])
          )
      - record: job:http_requests:rate2m
        expr: |
          sum by (job, status) (
            rate(http_requests_total[2m])
          )

The two recording rules give the dashboards a choice between [1 m] (responsive, noisier) and [2 m] (stable, slightly slower). Both are within the 2x to 4x range for a 15 s scrape.

Recording rules for SLO error budgets. A medium window for SLO calculation, evaluated every minute:

# /etc/prometheus/rules/slo.yaml
groups:
  - name: http-slo
    interval: 60s
    rules:
      - record: slo:http_requests:error_ratio_5m
        expr: |
          sum without (instance) (
            increase(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum without (instance) (
            increase(http_requests_total[5m])
          )

The [5 m] window is the SLO standard. It is stable enough to avoid single-scrape noise and short enough to detect a breach within 5 minutes.

Grafana panel settings. A panel for a 1-hour ops view:

  Panel type        : Time series
  Data source       : Prometheus
  Query             : job:http_requests:rate2m
  Min step          : 30s    # coarse enough to read, fine enough to be useful
  Legend            : show, by status
  Unit              : reqps

A panel for a 30-day SLO view:

  Panel type        : Time series
  Data source       : Prometheus
  Query             : slo:http_requests:error_ratio_5m
  Min step          : 5m
  Legend            : hide
  Unit              : percentunit (0 to 1)

The min step matches the SLO measurement step. A 30-day panel with min step=5 m asks for a value every 5 minutes, yielding ~8 640 data points across the range.

How to validate it

Three commands confirm the window is producing the expected behaviour.

Validate the window survives a scrape miss:

# READ-ONLY: query with a window that should survive one missed scrape
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[2m]))' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): a non-zero value even if the last scrape missed. The [2 m] window contains four samples; one missing scrape still leaves three.

Validate the window absorbs a counter reset:

# READ-ONLY: query a window that should span a known deploy
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
  --data-urlencode 'time=2026-08-13T10:30:00Z' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): a value close to the pre-deploy rate. The [5 m] window absorbs the rolling restart of a typical deploy; the rate across the window is the underlying traffic rate, not the reset spike.

Compare two windows to confirm the trade-off:

# READ-ONLY: compare [1m] and [5m] windows at a known spike
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[1m]))' \
  --data-urlencode 'time=2026-08-13T10:30:00Z' \
  | jq '.data.result[0].value[1]'
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
  --data-urlencode 'time=2026-08-13T10:30:00Z' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): the [1 m] value is higher than the [5 m] value if the spike is recent; the two values are within 10% if the spike has ended. A large discrepancy indicates a recent reset or a non-monotonic counter.

How it can fail

Six failure modes, each with a recognisable symptom:

1. Window shorter than 2x scrape interval. Too few samples; reset detection is unreliable. Symptom: panels with single-sample spikes that vanish when the window is widened.
2. Window much longer than the deploy cadence. A 1-hour window on a service that deploys every 10 minutes averages over multiple deploys. Symptom: a panel that shows the same value through the deploy; the deploy is invisible.
3. SLO window shorter than the SLO measurement step. A 1-minute rate window inside a 30-day SLO calculation produces a noisy SLO value. Symptom: SLO dashboards that oscillate on every step.
4. Different windows for the same metric on different dashboards. The HTTP error rate is shown with [1 m] on the ops dashboard and [5 m] on the SLO dashboard. The two panels disagree about whether the SLO is breached. Symptom: an incident review that finds the two panels tell different stories.
5. Window much longer than necessary. A 30-minute window on a 1-hour ops panel. A real incident at 02:55 is visible at 02:57, not 02:55. Symptom: an alert that fires after the user-visible degradation has ended.
6. Window too short for SLO consumption. A 1-minute rate window inside an SLO calculation that consumes a 5-minute window. The SLO calculation sums five 1-minute windows and averages over the variance. Symptom: an SLO that fires on noise but ignores a real breach.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Identify the question. “How fast just now”, “how fast in the last minute”, “how fast in the last hour”. The question dictates the window.
2. Confirm the scrape interval. scrape_interval in prometheus.yml. The window must be 2x to 4x this value.
3. Inspect the recording rule. The window is in the rule expression. Compare the rule’s window to the dashboard’s expectation.
4. Compare two windows. Run [1 m] and [5 m] at the same timestamp. A large discrepancy indicates a recent reset or a non-monotonic counter; identical values indicate the window is too long for the traffic shape.
5. Check the SLO measurement step. The window inside the SLO rule should match the SLO measurement step. A 5-minute SLO wants a 5-minute window.
6. Cross-check against application logs. A spike in the application log should produce a spike in the rate() panel at the same time. If the spike is missing, the window is smoothing it away.

Security implications

The window choice does not change the attack surface, but it changes the cost of a query:

• A user with query access can run rate() over arbitrary windows. A cardinality-bombing query over a 30-day window is more expensive than over a 5-minute window. Apply --query.max-concurrency and --query.timeout.
• Recording rules with long windows run on every Prometheus reload. The cost of a 30-day window is small per evaluation but unbounded in cardinality. Validate the rule with promtool check rules.
• The window does not protect the metric. The metric is on the wire regardless of the window. Lock the API behind authentication as usual.

Performance implications

The cost is dominated by the in-memory range vector at query time:

• Window length. Doubling the window doubles the in-memory range vector. A 30-day panel with [5 m] and 15 s scrapes keeps ~172 800 samples per series in memory. A 30 day panel with [30 m] keeps ~28 800.
• Step length. Halving the step doubles the number of evaluations. Net effect on memory: constant. Net effect on render time: doubled.
• Cardinality. A sum by (status) clause collapses per-instance series to one per status. Cardinality drops by an order of magnitude on a typical fleet.

Production guidance

• Default to [1 m] for ops dashboards, [5 m] for SLO error budgets, [15 m] for capacity trends. Document the choice in the recording rule.
• Use the same window across all panels that ask the same question. Different windows for the same metric is a smell.
• Validate the rule file with promtool check rules before reload. A malformed rule reloads successfully but evaluates to empty.
• Avoid windows shorter than 2x the scrape interval. Avoid windows much longer than the deploy cadence for ops panels.
• Pre-compute rates with recording rules. A 30-day panel should not re-evaluate rate() on every render.

Verification

You should now be able to answer:

• What is the 2x to 4x rule for the rate window relative to the scrape interval?
• Why is [5 m] the standard window for SLO error budgets?
• What is the relationship between the rate window and the deploy cadence?
• Why should the same metric use the same window across panels that ask the same question?
• What is the relationship between the rate window and the Grafana min step?

Quiz