irate() and increase()

A batch job runs once per hour. It increments a Prometheus counter batch_jobs_total by 1 on success, 0 on failure. On Grafana the panel rate(batch_jobs_total[1m]) shows nothing useful between runs; the rate is zero most of the minute and spikes for a single scrape when the job completes. The operator’s instinct is to widen the window. A 1-hour window hides the run entirely. The right answer is irate() for the panel, not rate() with a longer window.

A different question arrives the same morning: “how many requests did the edge proxy serve in the last five minutes?” Per-second is not what is asked. Total is asked. rate() over [5m] returns per second. Multiplying by 300 introduces rounding error. The right answer is increase().

This lesson covers the two functions that flank rate(): irate() for high-frequency response on low-volume counters, and increase() for total counts over a window.

What irate() and increase() are

irate(v[range_vector]) returns the per-second instant rate of change using only the last two samples in the range vector. The output is a rate; the window matters only because it must contain at least two samples. If the last two samples straddle a counter reset, irate() returns a negative number.

increase(v[range_vector]) returns the total increase of a counter over the window, in the same units as the counter itself. Internally it is rate(v[range_vector]) * (lastTs - firstTs). Reset handling is identical to rate().

Three-way contrast:

  Function    Window used       Output        Reset-aware?
  --------    -----------       ------        ------------
  rate()      full range        per-second    yes
  irate()     last 2 samples    per-second    no (returns negative)
  increase()  full range        total count   yes

Why a sysadmin cares

rate() is the default. irate() and increase() exist because rate() is the wrong tool in two recurring situations:

• Low-volume counters. A counter that increments once per hour, once per minute, or once per few minutes is invisible to rate() with a sensible window. rate() averages over too many “zero” samples. irate() looks at the last two samples and shows the burst when it happens.
• “How many” questions. Operations, finance, and capacity planning ask “how many requests in the last 5 minutes”, not “what was the per-second rate”. increase() returns the total directly, without the per-second-to-total mental arithmetic.

The wrong choice produces a dashboard that lies. A rate() panel on a low-volume counter shows a flat line that misses the event. A per-second panel on a “how many” question forces the reader to multiply by 300 and apologise for the rounding.

How irate() works

The mental model is the last two scrapes only. For samples (v[n-1], t[n-1]) and (v[n], t[n]):

irate  =  (v[n] - v[n-1]) / (t[n] - t[n-1])

No reset detection. No averaging across the window. Whatever happened between those two scrapes is the rate.

The two consequences:

1. Volatility. With a 15 s scrape interval, irate() on a normal counter looks like a sawtooth. Each scrape is its own data point. A panel that uses irate() never settles.
2. Counter resets produce negative rates. If a counter resets between the last two scrapes, v[n] less than v[n-1] and the output is negative. This is the dominant reason irate() is unsuitable for alerting.

How increase() works

The mental model is rate() scaled by the window duration:

increase  =  rate(v[range_vector]) * (lastTs - firstTs)

Internally the Prometheus engine computes rate() and multiplies by the wall-clock duration. Reset handling is therefore identical to rate(). The output is the counter’s own units (requests, bytes, errors) summed over the window.

Three properties worth remembering:

1. Multiplied by seconds. rate() returns per second. To get per minute, multiply by 60. To get the 5-minute total, multiply by 300. increase() does that multiplication internally and returns the answer in counter units.
2. Same reset behaviour as rate. A counter that resets inside the window is treated the same way: the missing increments are extrapolated. The increase() total therefore may be slightly higher than the raw counter delta because of the extrapolation.
3. Edge intervals are extrapolated. The first and last partial intervals inside the window are extrapolated to the window boundaries. A window of [4m] evaluated every 15 s covers the last 4 minutes plus a partial interval at the end. The partial interval adds a few seconds of estimated increase to the total.

Under the hood

How to configure it

Both functions are used inline in PromQL or inside recording rules. There is no daemon-level configuration. The configuration discipline is at the dashboard and rule level.

Recording rules for low-volume counters. A batch job that runs once per hour benefits from a recording rule that captures the per-run increment. Use increase() over a window that is slightly longer than the run cadence:

# /etc/prometheus/rules/batch.yaml
groups:
  - name: batch-jobs
    interval: 60s
    rules:
      - record: job:batch_jobs:increase1h
        expr: |
          increase(batch_jobs_total[1h])
      - record: job:batch_jobs:irate5m
        expr: |
          irate(batch_jobs_total[5m])

The recording rule job:batch_jobs:increase1h returns “how many jobs ran in the last hour” regardless of scrape jitter. The recording rule job:batch_jobs:irate5m returns “the per-second rate at the last two scrapes” for a Grafana panel that shows the burst.

Recording rules for SLO error budgets. For SLO calculations that ask “how many in 5 minutes”, use increase() inside the ratio:

# /etc/prometheus/rules/slo.yaml
groups:
  - name: http-slo
    interval: 30s
    rules:
      - record: slo:http_requests:error_ratio_5m
        expr: |
          sum without (instance) (
            increase(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum without (instance) (
            increase(http_requests_total[5m])
          )

The two increase() calls are over the same window and same label set, differing only in the status label filter. The ratio is the SLO error rate over the 5-minute window.

How to validate it

Three commands confirm the three functions are producing the expected outputs.

Validate rate() and irate() agree on a healthy counter:

# READ-ONLY: compare rate() and irate() on the same metric
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=rate(http_requests_total[2m])' \
  --data-urlencode 'time=2026-08-13T11:00:00Z' \
  | jq '.data.result[0].value[1]'

curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=irate(http_requests_total[2m])' \
  --data-urlencode 'time=2026-08-13T11:00:00Z' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative):

"412.8"
"418.5"

The two values are close but not identical. rate() averages the entire window; irate() reads the last two scrapes. A discrepancy of a few percent is normal. A discrepancy of 50% or more usually means the counter is non-monotonic, the scrape interval is much longer than expected, or the window spans a reset that irate() does not handle.

Validate increase() returns the expected total:

# READ-ONLY: increase() over 5 minutes should match the raw delta
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=increase(http_requests_total[5m])' \
  --data-urlencode 'time=2026-08-13T11:00:00Z' \
  | jq '.data.result[0].value[1]'

# Cross-check against the raw counter delta at the same instant
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=http_requests_total' \
  --data-urlencode 'time=2026-08-13T11:00:00Z' \
  | jq '.data.result[0].value[1]'
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=http_requests_total' \
  --data-urlencode 'time=2026-08-13T10:55:00Z' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): increase() returns ~124 000; the raw delta returns the same number, possibly slightly lower if a reset happened in the window. The increase() result may exceed the raw delta by a few percent because of the extrapolation at the edges.

Validate irate() against a low-volume counter:

# READ-ONLY: confirm irate() reports the burst on a sparse counter
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=irate(batch_jobs_total[5m])' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): a non-zero value in the seconds after the batch job completed, zero between runs. If the value is always zero, the window is missing the burst or the scrape interval is much longer than the job duration.

How it can fail

Six failure modes, each with a recognisable symptom:

1. irate() in an alert rule. Alert rules evaluate every 30 s to 1 m. irate() produces a fresh value every scrape. An alert based on irate() flaps on every scrape. Symptom: a constantly-firing-and-resolving Alertmanager notification.
2. increase() over a window that crosses a counter reset. increase() extrapolates the missing increments. The total is slightly higher than the raw delta. Symptom: a finance report that says “12 340 requests” when the raw counter delta shows 12 100.
3. increase() over a window longer than the metric lifetime. A counter that has only existed for 2 minutes cannot produce a meaningful increase(v[1h]). The evaluation returns an extrapolated value. Symptom: a “total in last hour” panel that wildly exceeds the counter’s actual lifetime.
4. irate() across a counter reset. irate() returns a negative number when the last two samples straddle a reset. Symptom: a panel that dips below zero whenever a rolling restart completes.
5. irate() inside sum() or avg() aggregations across many series. irate() is defined per-series. Summing irate() of different counters produces values without operational meaning. Symptom: a cluster-wide irate() panel that is dominated by the noisiest series.
6. rate() vs increase() unit confusion. Multiplying rate() by window seconds in a recording rule, then comparing against increase() output, produces a discrepancy caused by edge extrapolation. Symptom: the recording rule and the ad-hoc query disagree by a few percent.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Identify the question. “How fast”, “how fast just now”, or “how many”. The question dictates the function. Mixing the question and the function is the root of most dashboard bugs.
2. Check the window length. For irate(), the window only matters insofar as it must contain at least two samples. For increase(), the window is the period you are asking about. A wrong window produces a wrong answer.
3. Inspect the counter type. curl /metrics | grep TYPE. irate() and increase() assume a counter. A gauge input produces nonsense from both.
4. Test against a known event. For irate(), generate a burst (one increment on a low-volume counter) and confirm the panel spikes. For increase(), pick a known 5-minute interval, sum the raw counter delta, and confirm increase() matches within the extrapolation tolerance.
5. Check for aggregation. If the expression uses sum(), avg(), or any cross-series aggregation, confirm irate() is not being applied to series with different cadences.
6. Compare to a sibling panel. Plot rate(), irate() and increase() of the same counter side by side. They tell the same story in three voices.

Security implications

The three functions are query-time. The attack surface is the Prometheus API:

• irate() and rate() expose per-second business volume. increase() exposes per-window totals. Either way, a user with query access can reconstruct business activity from the counters. Lock the API behind authentication.
• irate() over a high-cardinality counter is a denial-of-service vector. The function is cheap per series, but cardinality-bombing queries apply. Apply --query.max-concurrency.
• Recording rules with irate() or increase() run on every Prometheus reload. A typo in a rule produces thousands of empty series per evaluation. Validate with promtool check rules.

Performance implications

• irate() is the cheapest of the three because it reads only two samples per series. rate() reads every sample in the window. increase() reads every sample and multiplies by seconds. The ranking: irate() cheapest, rate() and increase() similar.
• A 30-day SLO panel with rate() or increase() over [5m] uses ~8 640 samples per series in memory. The same panel with irate() uses 2 per series, but the answer is the instantaneous rate, not the SLO average. Choose the function that matches the question, then accept the cost.
• Window choice dominates cost. A [30d] window on a panel with 15 s scrapes uses 172 800 samples per series. That is the cost of long time ranges, not of any specific function.

Production guidance

• Pick the function from the question, not from habit. rate() is the default but not the only choice.
• Use irate() only on dashboards, never in alert rules.
• Use increase() when the answer is a total, not a rate.
• Use rate() inside SLO calculations; irate() produces unusable SLO values.
• For low-volume counters that scroll once per minute or longer, irate() over a window that catches the burst ([2m] or [5m]) is the only function that shows the event.

Verification

You should now be able to answer:

• Why is irate() unsuitable as the base of an alert rule?
• When should increase() be used instead of rate()?
• What happens to irate() output when the last two scrapes straddle a counter reset?
• Why does increase() return a slightly different number from the raw counter delta over the same window?
• Which of the three functions is cheapest at query time?

Quiz