Lab: Rate and Aggregation

Objective

Almost every number on an operational dashboard is rate() of something, summed over something. Both halves are easy to write and neither one announces when it is wrong, because the output of a wrong rate and the output of a right one are both a plausible-looking number in the right units.

This lab makes the correct answer computable in advance. You scrape three targets at three different intervals, and you pick a counter that increments exactly once per scrape — so the per-second rate of each target is exactly one divided by its scrape interval, and you know it before you ask. Every measurement after that is a comparison against a number you derived, not a number you hope is right.

Then you break one target and watch four functions that all claim to describe “how much this counter went up” produce four different answers, only two of which are usable.

Architecture

One Prometheus, three node_exporter containers, and — the only part that matters — three scrape configurations with three different intervals.

                     +-----------------------------+
                     |     prometheus 2.55  :9090  |
                     +--+------------+----------+---+
       scrape every 5s  |            | 15s      | 60s
                        v            v          v
                  +----------+  +---------+  +-----------+
                  | exp-edge |  | exp-core|  | exp-batch |
                  |   :9101  |  |  :9102  |  |   :9103   |
                  +----------+  +---------+  +-----------+
   expected rate     0.2/s        0.0667/s      0.0167/s
   of the scrape     = 1/5        = 1/15        = 1/60
   counter

The three exporters are identical. Nothing in this lab depends on what node_exporter measures; they are here because each one counts the requests made to its own /metrics endpoint, and Prometheus is the only thing making them. Set the scrape interval and you have set the counter’s rate.

Requirements

A Linux host with Docker Engine 28.x and Docker Compose v2, plus network access on first run to pull two images (roughly 300 MiB).
curl and jq on the host. Every number in this lab is read from the Prometheus HTTP API, never off a graph.
Free TCP ports 9090, 9101, 9102 and 9103.
Roughly 250 MiB of memory and a few hundred MiB of disk.
Time. Several measurements need a full five-minute rate window to have passed since the last change. Budget for the waiting; the numbers are wrong and confusing if you do not.
No out-of-band access requirement. The lab changes nothing outside its own directory and compose project.

Scenario

An on-call engineer is paged at 03:00 by RequestRateCollapsed, an alert built on sum(rate(http_requests_total[1m])) < 500. The panel behind the alert shows a deep V: traffic falls off a cliff at 02:51, sits near zero for a minute, and recovers to its previous level by 02:53.

Nothing was wrong. The 02:50 deployment rolled the fleet, every process restarted, and every counter went back to zero. But the alert did not fire during the previous four deployments, and the difference is not obvious: the panel expression was refactored last week from sum(rate(...)) to rate(sum(...)) by someone who was told the two were equivalent and reasonably assumed the cheaper-looking one was fine.

They are not equivalent, and this lab is where you find out how far apart they are — by causing the same reset deliberately, on a fleet whose correct answer you already know.

Tasks

Task 1: Build a fleet with three known rates

LABDIR="$HOME/rb-obs-rate"
mkdir -p "$LABDIR"
cd "$LABDIR"

prometheus.yml. The three scrape_interval values are the entire experiment design:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Fast. scrape_timeout must be <= scrape_interval or Prometheus
  # refuses to start; the default timeout is 10s, so 5s needs its own.
  - job_name: edge
    scrape_interval: 5s
    scrape_timeout: 4s
    static_configs:
      - targets: ['exp-edge:9100']

  # Inherits the global 15s.
  - job_name: core
    static_configs:
      - targets: ['exp-core:9100']

  # Slow. Task 4 depends on this one being slower than the windows
  # people habitually type.
  - job_name: batch
    scrape_interval: 60s
    static_configs:
      - targets: ['exp-batch:9100']

compose.yaml:

name: rb-obs-rate

x-exporter: &exporter
  image: prom/node-exporter:v1.8.2
  restart: unless-stopped

services:
  exp-edge:
    <<: *exporter
    container_name: rb-rate-exp-edge
    ports:
      - '9101:9100'

  exp-core:
    <<: *exporter
    container_name: rb-rate-exp-core
    ports:
      - '9102:9100'

  exp-batch:
    <<: *exporter
    container_name: rb-rate-exp-batch
    ports:
      - '9103:9100'

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-rate-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=3h'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'
    depends_on:
      - exp-edge
      - exp-core
      - exp-batch

volumes:
  prom-data:

Service impact possiblelab host— Starts four containers and binds host ports 9090 and 9101-9103

$ docker compose up -d

curl -sf http://localhost:9090/-/ready && echo PROMETHEUS-READY

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
| jq -r '.data.result[] | "\(.metric.job)\tup=\(.value[1])"' | sort

Three jobs, all up=1.

Task 2: Establish the counter’s contract

Do not take on trust that this counter increments once per scrape. Prove it, now, while the stack warms up. Pick the slowest target, so a Prometheus scrape is unlikely to land between your two reads:

for i in 1 2; do
  V=$(curl -s http://localhost:9103/metrics \
      | awk '/^promhttp_metric_handler_requests_total\{code="200"\}/ {print $2}')
  echo "read $i: $V"
  sleep 2
done

The second read is exactly one higher than the first, and the increment came from your own first request. The handler counts every GET of /metrics, and a scrape is nothing more than a GET. That is the contract this whole lab rests on, and it also means the observer is part of the system: every direct curl in this lab perturbs the rate on the target it reads.

Confirm the metric’s declared type while you are here, because rate() assumes it:

curl -s http://localhost:9103/metrics | grep '^# TYPE promhttp_metric_handler_requests_total'

A line ending in counter is the contract. Now leave the stack alone for six minutes — the direct reads you just made are inside the next window, and they will bend the numbers in Task 3 by exactly as much as you would expect them to.

Task 3: Why the counter is unreadable and the rate is not

Look at the raw counter first. This is what a panel plots when someone forgets the rate():

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=promhttp_metric_handler_requests_total{code="200"}' \
| jq -r '.data.result[] | "\(.metric.job)\t\(.value[1])"' | sort

Three integers that are not comparable to one another. Each encodes how long that process has been running and how often it is scraped, mixed together, with no way to separate the two. Nothing operational can be read off those numbers: not “is traffic up”, not “is one target quieter than the others”, not “did anything change”. The only thing a raw counter supports is a comparison with itself at another point in time, which is precisely what rate() does for you.

Now the rate, against the number you already know it should be:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(promhttp_metric_handler_requests_total{code="200"}[5m])' \
| jq -r '.data.result[] | "\(.metric.job)\t\(.value[1])"' | sort

Job	Scrape interval	Expected rate	Meaning
`edge`	5s	0.2 /s	one scrape every five seconds
`core`	15s	0.0667 /s	one scrape every fifteen seconds
`batch`	60s	0.0167 /s	one scrape every sixty seconds

Each measured value should land within a fraction of a percent of 1/interval. Ask the engine to check it for you rather than eyeballing three decimal expansions:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(promhttp_metric_handler_requests_total{code="200",job="batch"}[5m]) * 60' \
| jq -r '"scrapes per minute on the batch job: \(.data.result[0].value[1])"'

One. The unit of rate() is always per second, which is why every dashboard that wants “per minute” carries a * 60 and every dashboard that forgot it is off by sixty.

Why the rate is exactly 1/interval and not slightly under

The samples inside a five-minute window do not reach its edges. At a 60-second interval a [5m] window holds five samples spanning roughly 240 seconds, so the counter is observed to rise by four — and four increments over 300 seconds would be 0.0133/s, not 0.0167/s.

rate() does not do that arithmetic. It measures the rise across the samples it actually has, then extrapolates that slope out to both edges of the window: 4 increments over 240 seconds becomes 5 over 300. The extrapolation is capped — if the gap to a window edge is much larger than the typical gap between samples, rate() extends only half a sample interval rather than inventing data across a hole — but for a healthy target scraped on schedule, the effect is to return the slope you would have measured with perfect boundary alignment.

That is why the answer is 1/interval rather than (N-1)/window, and it is also the mechanism behind the non-integer increase() values in Task 5.

Task 4: Where rate() returns nothing at all

rate() needs at least two samples inside the window to have any rise to measure. Sweep the window across all three jobs and find the floor:

for W in 10s 15s 30s 1m 2m 5m; do
  printf 'window=%-4s ' "$W"
  curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode "query=rate(promhttp_metric_handler_requests_total{code=\"200\"}[$W])" \
  | jq -r '[.data.result[] | .metric.job] | sort | join(",") | if . == "" then "(no result)" else . end'
done

Read the shape of the output rather than the individual rows. edge appears at every window, because five seconds fits twice into ten. core disappears at 10s. batch does not appear until the window exceeds sixty seconds, and is only reliable well beyond that — a [1m] window on a 60-second scrape holds one or two samples depending on where the evaluation instant falls, so it flickers between a number and nothing.

The operational rule that falls out: make the window at least four times the scrape interval. Two samples is the arithmetic minimum and a single missed scrape takes you below it; four gives the query somewhere to fail from.

What if this query were behind an alert instead of a panel?

The empty result is the dangerous half of this task. rate() did not error and Prometheus did not log anything — the expression evaluated correctly and matched no samples, which is a legitimate outcome.

A Grafana panel renders that as “No data”, visually identical to a target that has stopped existing. An alerting rule whose expression returns an empty vector does not fire, because there is no series to compare against a threshold: an alert written as rate(errors[1m]) > 10 against a metric scraped every 60 seconds is permanently silent, and it is silent in exactly the way a healthy system is silent. It will pass every review, sit in the rules file for a year, and never once tell you anything.

The check that catches it is absent() beside every threshold rule, or the discipline of asserting in CI that each alert expression returns a series against a live TSDB.

Task 5: increase(), and why it is not a whole number

increase(v[w]) is defined as the rise of the counter across the window. On a counter that only ever goes up by one, over a window that fits a whole number of scrape intervals, you would expect a whole number:

for W in 5m 100s; do
  printf 'increase[%s]\n' "$W"
  curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode "query=increase(promhttp_metric_handler_requests_total{code=\"200\"}[$W])" \
  | jq -r '.data.result[] | "  \(.metric.job)\t\(.value[1])"' | sort
done

The [5m] row lands on or very near 60, 20 and 5 — the window divided by each job’s interval, which is what the extrapolation in Task 3 produces. The [100s] row does not land on anything tidy, particularly for batch, where a 100-second window holds two samples spanning 60 seconds and the slope between them is stretched to cover 100.

increase() is rate() multiplied by the window; it inherits the extrapolation and therefore returns a real number, not a count of events. Reporting it as “requests in the last five minutes” is fine. Reporting increase(orders[24h]) as “orders today” on a finance dashboard is not, and the discrepancy will be found by someone who reconciles it against the database.

Task 6: Aggregate a fleet whose total you already know

Three rates, all known: 0.2, 0.0667 and 0.0167. Their sum is 0.2833/s. Confirm it, then look at what each aggregator does to the label set:

BASE='rate(promhttp_metric_handler_requests_total{code="200"}[5m])'

for Q in "sum($BASE)" "avg($BASE)" "count($BASE)" "max($BASE)" "min($BASE)"; do
  printf '%-12s ' "${Q%%(*}"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[0].value[1]'
done

sum is 0.2833 and answers “how many scrapes per second is this Prometheus performing” — a real question with a real answer. avg is 0.0944 and answers nothing: no target runs at that rate, and the number moves whenever a target is added or removed regardless of whether any traffic changed. It is the mean of a distribution nobody drew.

Now the label question. sum with no clause collapsed everything to one series with no labels at all. Get the dimensions back:

BASE='rate(promhttp_metric_handler_requests_total{code="200"}[5m])'

curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=sum by (job) ($BASE)" \
| jq -r '.data.result[] | "by(job)      \(.metric | tostring)\t\(.value[1])"' | sort

curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=sum without (code) ($BASE)" \
| jq -r '.data.result[] | "without(code) \(.metric | tostring)"' | sort

by (job) keeps exactly the labels it names and drops every other one, including instance and code. without (code) keeps everything except the one it names, so job and instance both survive. On a three-target fleet the difference is cosmetic; in a rule file it decides whether a new label added upstream next quarter silently changes the shape of your output series. by breaks loudly when a dimension you named disappears; without absorbs dimensions you never asked for.

Finally, the aggregator that answers “which one”:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=topk(1, sum by (job) (rate(promhttp_metric_handler_requests_total{code="200"}[5m])))' \
| jq -r '.data.result[] | "busiest: \(.metric.job) at \(.value[1])/s"'

edge, at 0.2/s. topk keeps the labels of the winners, which is what makes it useful in an alert annotation and what makes it expensive: it must compute every series before it can rank them.

Task 7: Reset a counter and watch four functions disagree

This is the scenario. Record the current fleet total first, so you have a before:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(promhttp_metric_handler_requests_total{code="200"}[5m]))' \
| jq -r '"before: \(.data.result[0].value[1]) scrapes/s"'

Service impact possiblelab host— Restarts one exporter; its counters reset to zero

$ cd ~/rb-obs-rate && docker compose restart exp-core

Wait about ninety seconds, then look at the raw counter across the restart:

END=$(date -u +%s)
START=$((END - 300))
curl -sG http://localhost:9090/api/v1/query_range \
  --data-urlencode 'query=promhttp_metric_handler_requests_total{code="200",job="core"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" --data-urlencode 'step=15s' \
| jq -r '.data.result[0].values[] | "\(.[0])\t\(.[1])"' | tail -20

A cliff: a large number, then a small one, then counting up again from near zero. Now ask four different functions what happened during that window:

S='promhttp_metric_handler_requests_total{code="200",job="core"}'

for F in "rate($S[5m])" "increase($S[5m])" "delta($S[5m])" "irate($S[5m])"; do
  printf '%-14s ' "${F%%(*}"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$F" \
  | jq -r '.data.result[0].value[1] // "(no result)"'
done

rate is close to 0.0667 — a little under, because scrapes were genuinely missed while the container restarted, but nothing like a cliff. It saw the value go backwards, concluded that a counter cannot do that, and treated the drop as a reset rather than as negative traffic.
increase is that rate times 300, and is corrected the same way.
delta is a large negative number. delta() has no reset handling at all: it is the last value minus the first, extrapolated, and it is documented for gauges. Pointed at a counter it reports the restart as an enormous decrease in traffic.
irate has already recovered to about 0.0667 regardless of the restart, because it looks only at the final two samples in the window. That is its virtue and its vice: it recovers instantly and it is blind to everything older than one scrape interval.

Now the expression from the scenario. rate() needs a range vector, and sum() returns an instant vector, so rate(sum(x)[5m]) cannot be written directly — you need a subquery to make one:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(promhttp_metric_handler_requests_total{code="200"}[5m]))' \
| jq -r '"sum(rate(...))  = \(.data.result[0].value[1])"'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(sum(promhttp_metric_handler_requests_total{code="200"})[5m:15s])' \
| jq -r '"rate(sum(...))  = \(.data.result[0].value[1])"'

The two numbers disagree, and the second one is wrong. Here is why, and it is worth being precise about because the rule generalises:

sum() of three counters produces a series that is not a counter. It is a number that happens to be increasing right up until one of its inputs restarts, at which point it steps down by that input’s entire accumulated value — a step that has nothing to do with the rate of anything. rate() then finds a decrease, applies the only interpretation it has, and adds the pre-drop value back into the numerator. A restart that cost a handful of scrapes is reported as thousands of events.

sum(rate(x)) never has this problem because each rate() is computed against a series that really is a counter, and only then are the results — which are plain per-second numbers, freely addable — summed.

Task 8: Watch the fleet total hide a dead target

One more failure the sum will not tell you about.

Service impact possiblelab host— Stops one exporter; Prometheus keeps scraping and failing

$ cd ~/rb-obs-rate && docker compose stop exp-batch

Wait five minutes so the window contains only post-failure scrapes, then:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(promhttp_metric_handler_requests_total{code="200"}[5m]))' \
| jq -r '"fleet total now: \(.data.result[0].value[1]) scrapes/s"'

It fell from 0.2833 to roughly 0.2667: a six percent move, on a fleet that just lost a third of its targets. No threshold anyone would set on a traffic metric catches a six percent dip, and no human watching the panel notices one. The aggregation did not fail — it faithfully summed what was there, and what was there was smaller.

The query that does notice is not a rate query at all:

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up == 0' \
| jq -r '.data.result[] | "DOWN: job=\(.metric.job) instance=\(.metric.instance)"'

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=count(up) - count(up == 1)' \
| jq -r '"targets down: \(.data.result[0].value[1])"'

Restart it before moving on:

docker compose start exp-batch

Validation

Four checks. Each one proves a claim rather than repeating a step.

1. Every measured rate equals one over its scrape interval. Invert the rate and you should get the configured interval back, in seconds:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=1 / rate(promhttp_metric_handler_requests_total{code="200"}[5m])' \
| jq -r '.data.result[] | "\(.metric.job)\tseconds between scrapes = \(.value[1])"' | sort

Approximately 5, 15 and 60 — the three intervals from prometheus.yml, recovered from the metric rather than read back off the config. If one of them is materially low, something other than Prometheus is reading that exporter.

2. The floor on the rate window is real and predictable. The slowest job returns nothing at a window it cannot fill:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(promhttp_metric_handler_requests_total{code="200",job="batch"}[30s])' \
| jq -r '"result count at [30s]: \(.data.result | length)"'

Zero, with status still success.

3. sum(rate(x)) and rate(sum(x)) disagree across the reset you caused. Both numbers, side by side, with the difference:

C='promhttp_metric_handler_requests_total{code="200"}'
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode "query=rate(sum($C)[5m:15s]) - sum(rate($C[5m]))" \
| jq -r '"error introduced by aggregating before rating: \(.data.result[0].value[1])"'

Run it inside five minutes of the restart. A number far from zero is the bug in the scenario, measured.

4. The fleet-wide sum is not a target-health signal. Record the percentage the sum moved when a third of the fleet died, and confirm up caught it when the sum did not. Both numbers belong in your lab notes.

Expected Outcome

Four containers running: one Prometheus scraping three targets at 5s, 15s and 60s.
A table of three measured rates, each matching 1 / scrape_interval, and the inverted query that recovers the interval from the metric.
The window at which rate() stops returning a result for the 60-second job, with the two-sample arithmetic that predicts it.
Four different answers to “what did this counter do across the restart”, with a stated reason to trust rate and increase, to use irate knowingly, and never to point delta at a counter.
A measured disagreement between sum(rate(x)) and rate(sum(x)) across a reset, and the sentence that explains it: a sum of counters is not a counter.
A recorded measurement of how little a fleet total moves when a target dies.

Troubleshooting

Prometheus exits immediately with a message about scrape timeout. The edge job sets a 5-second interval; the default scrape_timeout is 10 seconds, and a timeout longer than the interval is rejected at config load. The scrape_timeout: 4s line in the edge job is not optional.
rate() returns nothing for a job you expected. The window holds fewer than two samples. Multiply the job’s scrape interval by four and use that as the minimum window.
The measured rate is above 1/interval. Something else is requesting /metrics. Your own curl from Task 2 counts, and so does a browser tab left open on the exporter. Wait one full window after any direct read.
promhttp_metric_handler_requests_total is not in the exposition. node_exporter includes its own process metrics by default; --web.disable-exporter-metrics removes them. This lab’s compose file passes no flags, so the metric is present.
The numbers after the restart look nothing like the table. The five-minute window still contains pre-restart samples. Wait a full five minutes after any restart before comparing against a steady-state value.
query_range returns an empty values array. start and end must be Unix timestamps or RFC 3339, and start must be inside the retention window. The date -u +%s form in Task 7 produces the right thing on both GNU and BSD date.

Cleanup

Everything the lab created is one directory, one compose project and one named volume.

Data-loss risklab host— Removes the four containers and this project's Prometheus volume

$ cd ~/rb-obs-rate && docker compose down -v

docker volume ls | grep rb-obs-rate || echo "volumes gone"
rm -rf "$HOME/rb-obs-rate"

No host networking, firewall or filesystem state outside the lab directory was changed, so there is nothing else to restore.

Production notes

Rate first, aggregate second — always. sum(rate(x[5m])) is correct and rate(sum(x)[5m:15s]) is not, for a reason that has nothing to do with cost: reset detection only works while the series is still one process’s counter. Make this a review rule, because the wrong form is not visibly wrong and produces the right answer whenever nothing is restarting.

Set the rate window from the scrape interval, not from habit. Four times the scrape interval is the working floor: two samples is the arithmetic minimum and one missed scrape takes you below it. At the common 15-second scrape, that puts [1m] at the edge and [2m] in comfortable territory; [5m] is the usual choice for alerting because it also survives a rolling restart.

Every threshold rule needs a companion that fires on absence. An expression that returns no series does not fire, so a rate alert whose window is too short is permanently silent and indistinguishable from a healthy service:

groups:
  - name: traffic
    rules:
      - alert: RequestRateCollapsed
        expr: sum by (job) (rate(http_requests_total[5m])) < 500
        for: 10m
        labels:
          severity: warning
      # The rule above cannot fire if the metric stops existing.
      - alert: RequestMetricAbsent
        expr: absent(sum by (job) (rate(http_requests_total[5m])))
        for: 10m
        labels:
          severity: warning

Never point delta() at a counter, and be deliberate about irate(). delta() has no reset handling and is documented for gauges. irate() uses only the last two samples, which makes it the right tool for a high-resolution view of a fast-moving signal and the wrong tool for an alert, where its sensitivity to a single scrape is a false-positive generator.

A fleet total is not a health signal. Losing a third of a fleet moved the sum in this lab by six percent. Alert on up == 0, on absent(), and on target count against an expected value; use aggregate traffic for capacity conversations, not for detection.

State the units in the panel title. rate() is per second, everywhere, with no unit metadata anywhere in the query. Every “the dashboard was off by 60” incident is a per-second number labelled per-minute, and the title is where that gets caught.

What You Learned

A counter’s per-second rate is computable in advance if you control its schedule. Three targets, three intervals, three rates equal to 1 / interval — measured, not assumed.
rate() extrapolates to the window edges, which is why the answer is 1/interval rather than (samples-1)/window, and why increase() over a window is rarely a whole number of events.
Below two samples, rate() returns an empty vector and says nothing. On a panel that is “No data”; on an alert it is permanent silence that looks exactly like health.
delta, irate, increase and rate gave four different answers to the same restart. Only two of them were usable, and the difference is entirely about which ones know a counter can reset.
A sum of counters is not a counter. Aggregating before rating destroys the property rate() depends on, and the resulting error appears only during the restarts the query most needs to survive.
The fleet total moved six percent when a third of the fleet died. Detection belongs to up and absent(); aggregate traffic is for capacity.

Objectives

Prerequisites