rate() and Counter Resets

A 03:00 page lands. Prometheus is alerting on rate(http_requests_total[1m]) > 1000. The on-call opens Grafana. The panel shows a clean spike to roughly 2 000 req/s starting at 02:55 and dropping back to 1 000 req/s by 02:57. The deployment log shows a rolling restart of the application fleet between 02:50 and 02:56. Every pod restart reset http_requests_total to zero. rate() detected each reset and extrapolated the missing increments across the reset window. The spike in the panel is the cost of rolling restarts on a 1-minute rate window. The on-call has to read the panel correctly to recognise that the spike is a deployment artefact, not a real traffic anomaly.

This lesson is about reading rate() correctly. The function is load-bearing for almost every Grafana panel, alert rule and recording rule in a Prometheus platform. Misunderstanding it is the single most common source of false-positive alerts and “the graph looks weird” tickets.

What rate() is

rate(v[range_vector]) returns the per-second average rate of change of a counter over the window. The expression rate(http_requests_total[1m]) answers one question: “how many HTTP requests per second did this counter accumulate over the last minute, on average?”

Three properties matter:

1. Counter-centric. rate() assumes the input is a monotonic counter. It detects resets (drops in value) and extrapolates over them. On a non-counter that happens to decrease, rate() treats the decrease as a reset and produces nonsense.
2. Per-second. The unit is always per second. A rate over a 1-minute window of 6 000 requests returns 100, not 6 000.
3. Average, not instantaneous. rate() returns the mean rate across the window. It does not return the current second. For the current-second view, use irate() (next lesson) or a histogram_quantile over a heatmap.

The alternative is to plot the raw counter. The next lesson explains why that fails. For now, accept that almost every useful PromQL panel starts with rate().

Why a sysadmin cares

Three operational pains disappear once rate() is understood:

• “How busy is the service right now?” answered in one query, one Grafana panel, one alert rule.
• “Did the deploy increase the request rate?” answered by comparing rate() before and after the change. Raw counter values cannot be compared because their starting points differ.
• “Is the cluster healthy after the rolling restart?” answered despite resets. rate() is the only PromQL function that recovers the trend line through counter resets without special-case logic.

The pain of not understanding rate() shows up at 03:00. A suspicious spike or dip appears. The engineer does not know whether the spike is real, whether the rate window is too short, whether the counter is non-monotonic, or whether a scrape missed. The investigation stalls. The lesson exists to make that stall shorter.

How rate() works

The mental model is a sequence of samples with timestamps. For the series {v_0, v_1, ..., v_n} at times {t_0, t_1, ..., t_n} inside the range, rate() computes:

                    v_n - v_0 + sum of reset extrapolations
rate  =  --------------------------------------------------------
                                  t_n - t_0

Where sum of reset extrapolations is the value Prometheus adds to the numerator to account for each detected counter reset. A reset is detected between two consecutive samples whenever v_{i+1} < v_i. For each detected reset, Prometheus extrapolates the value the counter would have had if it had continued increasing at the average rate of the surrounding samples.

The extrapolation is conservative. Prometheus assumes the counter kept increasing at the same rate it had before the reset. The rate across the reset window is therefore indistinguishable from the rate just before and just after it.

Two edge cases dominate the production calls:

• Two samples only. rate() returns (v_1 - v_0) / (t_1 - t_0) with no reset detection (there is no pair to compare for a reset). This is the dominant case at short windows on default-scrape metrics.
• One or zero samples. rate() returns no result. The expression evaluates to an empty vector and the panel shows “No data”. This is what happens when the rate window is shorter than the scrape interval.

The diagram below shows a counter, two resets, and the rate that rate() produces from it:

value
   ^
   |                                sample n
   |                              /
   |            sample 1        /
   |            /             /
   |          /             /
   |        /   reset     /
   |      /     v      /
   |    /          /
   |  /         /
   |/________/__________________________ time
   sample 0  t1   t2     t3         tn

   rate() across tn - t0 returns
       (vn - v0 + 2 resets) / (tn - t0)

The line rate() across tn - t0 is the average across the whole window. It is the value the Grafana panel shows.

Under the hood

How to configure it

Three layers. The first is the instrumentation library or exporter that produces the counter. The second is the Prometheus configuration that scrapes it. The third is the PromQL expression that uses it.

Layer 1: exporter / instrumentation. The Prometheus Go client defaults to monotonic counters. Node exporter exposes node_network_*_bytes_total as a monotonic counter that resets on process restart (no exporter runs forever). Verify the metric is reported as a counter:

# READ-ONLY: confirm the metric type in the scrape output
curl -sf http://node-exporter:9100/metrics \
  | grep -E '^# (TYPE|HELP) node_network_receive_bytes_total'

Expected output (illustrative):

# HELP node_network_receive_bytes_total Network device statistic receive_bytes.
# TYPE node_network_receive_bytes_total counter

A line beginning # TYPE ... counter is the contract. If the line says gauge, the metric cannot be used with rate() safely.

Layer 2: Prometheus scrape config. The default scrape_interval: 15s is fine for most counters. Lower it for high-frequency business metrics (1 s or 5 s) only if the cardinality justifies it:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s   # production default; do not lower without cost review

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_network_(receive|transmit)_bytes_total'
        action: keep

Layer 3: PromQL expression. A recording rule is the production-grade way to share the expression across dashboards and alerts:

# /etc/prometheus/rules/network.yaml
groups:
  - name: network-rate
    interval: 30s           # CONFIGURATION: rule evaluation cadence
    rules:
      - record: instance:node_network_receive_bytes:rate2m
        expr: |
          rate(node_network_receive_bytes_total[2m])
        labels:
          severity: info
      - record: cluster:node_network_receive_bytes:rate2m
        expr: |
          sum without (instance, job) (
            rate(node_network_receive_bytes_total[2m])
          )

The 2-minute window is roughly 8x the 15 s scrape interval, well inside the 2x to 8x range that gives a stable average without smoothing incidents away.

How to validate it

Three commands confirm the expression is live, returning the expected shape, and behaving correctly across a counter reset.

Confirm the expression is parseable and live:

# READ-ONLY: instant query at "now"
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=rate(node_network_receive_bytes_total[2m])' \
  | jq '.data.result | length'

Expected output (illustrative):

12

A positive integer equal to the number of scraped instances. Zero means the metric is not being scraped, the label selector is wrong, or Prometheus has no targets up.

Inspect a single series in detail:

# READ-ONLY: show metric + labels + value for one series
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=rate(node_network_receive_bytes_total[2m])' \
  --data-urlencode 'time=2026-08-13T10:00:00Z' \
  | jq '.data.result[0]'

Expected output (illustrative):

{
  "metric": {
    "__name__": "node_network_receive_bytes_total",
    "instance": "node-01.internal:9100",
    "device": "eth0",
    "job": "node"
  },
  "value": [
    1755074400,
    "84523.6"
  ]
}

The value is a per-second rate. 84 523.6 bytes per second is roughly 660 kbit/s. If the panel shows raw bytes, the rate() expression is missing in the Grafana query.

Validate across a counter reset:

# READ-ONLY: range query that includes a known pod restart
curl -sG http://prometheus:9090/api/v1/query_range \
  --data-urlencode 'query=rate(node_network_receive_bytes_total[2m])' \
  --data-urlencode 'start=2026-08-13T09:55:00Z' \
  --data-urlencode 'end=2026-08-13T10:05:00Z' \
  --data-urlencode 'step=15s' \
  | jq '.data.result[0].values[]'

Expected output (illustrative): a smooth list of [timestamp, value] pairs with no spikes above the 99th percentile of the pre-restart rate. If a single value spikes tenfold, the rate window is too short and a reset is leaking through extrapolation.

How it can fail

Six failure modes, each with a recognisable symptom:

1. Non-monotonic counter. A library reports “current queue depth” as a counter. Every time the queue empties, the value drops. rate() sees the drop as a reset and extrapolates a huge increase. Symptom: a panel that spikes by 10x whenever the queue empties.
2. Window too short. rate(v[15s]) on a 15 s scrape interval has exactly one or two samples per evaluation. Reset detection has too few pairs; extrapolation can fire on noise. Symptom: panels with single-sample spikes that disappear when the window is widened to 2m.
3. Window too long. rate(v[30m]) smooths over real incidents. A 30-second spike in error rate that breaks the SLO is invisible. Symptom: alerts that fire only after the incident is already visible to users.
4. Scrapes missed. A scrape failure means the range vector has fewer samples than expected. rate() extrapolates over the gap. Symptom: spurious low values when a scrape misses (rate treats the gap as zero increase) followed by a spike when the next scrape arrives.
5. Counter that never increments. A metric that does not change between scrapes still appears in the range vector, but with two equal samples. rate() returns zero. Symptom: zero-rate panels for counters that should never be zero.
6. Counter resets faster than the scrape interval. A short-lived batch job that runs and resets in under 15 s produces a single sample per execution. rate() cannot detect the reset because there is no second sample to compare against. Symptom: panel shows the cumulative increase of the last batch execution, not a per-second rate.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Confirm the metric is a counter. curl the /metrics endpoint, grep for the metric, confirm # TYPE ... counter. If it is gauge, the panel cannot be fixed at the PromQL layer; fix the instrumentation.

2. Confirm Prometheus is scraping it. up{job="node"} is the fastest check. A value of 0 means the target is down.

3. Look at the raw counter. Plot node_network_receive_bytes_total directly. A monotonically rising line with occasional drops is a counter with resets; a line that goes up and down is a mislabelled gauge.

4. Inspect the rate window. Open the panel in Grafana and read the min step and the query editor. A min step of 15 s against a rate window of 15 s gives one to two samples per step. Widen the rate window or shorten the min step.

5. Test the expression in isolation. Use promtool query instant against the production endpoint:

   promtool query instant http://prometheus:9090 \
     'rate(node_network_receive_bytes_total[2m])'

6. Check for known resets. Cross-reference spike times against the Kubernetes rollout log or the systemd journal. A spike that aligns with every rolling restart is a reset, not a real rate increase.

Security implications

rate() is a query-time function. It does not introduce new attack surface; it consumes the same Prometheus API surface as every other query. The relevant risks live at the boundaries:

• The Prometheus HTTP API is unauthenticated by default. A query like rate(http_requests_total[1m]) exposes the per-second business volume to anyone who can reach the API. Lock the API behind a reverse proxy with basic auth or mTLS.
• A user with query access can run rate() over arbitrary range vectors. Cardinality-bombing queries (rate({__name__=~".+"}[5m])) are a denial-of-service vector. Apply --query.max-concurrency and --storage.tsdb.retention to bound the cost.
• Recording rules that contain rate() run on every Prometheus reload. A typo in a recording rule produces thousands of empty series per scrape interval. Validate the rule with promtool check rules before applying.

Performance implications

rate() is cheap per call. Its cost is the in-memory range vector, which is dominated by the window size relative to the scrape interval. A 5-minute window against a 15 s scrape interval keeps 20 samples per series per evaluation. A 30-day SLO panel keeps 172 800 samples per series in memory and computes rate() once per step. That is the cost the platform pays for long time ranges; it is not specific to rate().

The levers:

• Window length. Doubling the window doubles the in-memory range vector. A 30-day panel with 5m steps uses 8 640 samples per series; the same panel with 1m steps uses 43 200.
• Step length. Halving the step halves the per-step cost but doubles the number of steps. Net effect on query CPU: roughly constant; net effect on memory: constant; net effect on render time: doubled.
• Cardinality. rate() does not change cardinality. A recording rule that sums before applying rate() reduces it.

Production guidance

• Treat rate windows as code. Store the chosen window in the Prometheus rule file and review it in code review.
• Pick the window to match the cadence of the events you care about: [1m] for traffic-shape panels, [5m] for SLO error budgets, [15m] for capacity-planning trends.
• Always plot rate() over a label-matched metric. A raw counter on a dashboard is almost always wrong.
• Use recording rules for any rate() expression that appears in more than one dashboard or alert. Recording rules make evaluation cost predictable.
• Validate every rule file with promtool check rules before reload. A malformed rule reloads successfully but evaluates to empty.

Verification

You should now be able to answer:

• What does rate() return when the range vector contains exactly two samples?
• What does rate() do when it detects a decrease between consecutive samples?
• What happens when rate() is applied to a metric that is declared as a gauge?
• What is the relationship between the rate window and the scrape interval in production?
• How would you validate a rate() expression against a known counter reset?

Quiz