Aggregation Across Instances

A Tuesday deploy leaves one of twenty application instances with a broken connection pool. That instance returns errors at 40 per second; the other nineteen sit near zero. The fleet-average error panel shows 2 per second — inside the normal band, painted green. The per-instance panel, had anyone opened it, shows one bright red line. Aggregation decided which of those two stories the monitoring told, and nobody chose the aggregation deliberately.

Aggregating across instances is the bread and butter of PromQL: taking many time series and collapsing them into the few numbers a human or an alert can act on. Doing it well means keeping the labels an investigation needs and choosing an operator whose failure shape you understand.

What it is

PromQL aggregation operators take an instant vector — many series, one sample each — and produce a smaller instant vector. The operators you will use daily:

• sum — add the values. Fleet throughput, total error count.
• avg — the mean. Utilisation across a fleet.
• min / max — the extremes. The emptiest disk, the hottest host.
• count — how many series exist. How many targets are up.
• topk / bottomk — the k largest or smallest series, kept as individual series rather than collapsed.
• quantile — a quantile over the values in the group.
• stddev / stdvar — spread. Useful for spotting divergence inside a group.
• group — collapses each group to the constant 1. Used to test existence, mostly in joins.
• count_values — builds a frequency count of sample values.

Grouping is controlled by a clause:

sum by (job) (rate(http_requests_total[5m]))        # keep only job
sum without (instance) (rate(http_requests_total[5m]))  # drop instance

by lists the labels the result keeps; everything else is dropped. without lists the labels to drop; everything else is kept. The two forms answer different questions: by when you know exactly what you want to group on, without when you know exactly what you want to remove.

Why a sysadmin cares

Two operational views fight for the same metrics:

• Fleet view. “Is the service healthy?” — needs aggregation. Twenty per-instance lines are unreadable on one panel, and an alert per instance pages twenty times for one fleet event.
• Instance view. “Which instance is sick?” — needs the labels aggregation throws away. The drill-down that finds the culprit depends on instance, pod or host still being present.

Every aggregation you write picks a point between those views. Pick wrong and you get one of two incident shapes: a sick instance hidden inside a healthy average, or a wall of per-instance alerts with no fleet signal. The skill is choosing the operator and the label set that match the question — and recording both views so neither is lost.

How it works

Aggregation collapses the label set to exactly what the clause names:

Input series (rate already applied):
  {job="api", instance="10.0.0.11:9090", route="/pay"}   12
  {job="api", instance="10.0.0.12:9090", route="/pay"}   10
  {job="api", instance="10.0.0.11:9090", route="/cart"}   8
  {job="api", instance="10.0.0.12:9090", route="/cart"}   9

sum by (job):
  {job="api"}   39            <- instance and route are gone

sum by (job, route):
  {job="api", route="/pay"}   22
  {job="api", route="/cart"}  17   <- instance is gone

sum without (instance):
  {job="api", route="/pay"}   22
  {job="api", route="/cart"}  17   <- same result, opposite clause

The output series identity is built only from the surviving labels. This matters for alerts: an alert on sum by (job) cannot tell you which instance is burning, because that information no longer exists in the result.

The canonical production pattern pairs a fleet aggregate with a per-instance view:

# Fleet request rate, one series per job
sum by (job) (rate(http_requests_total[5m]))

# Per-instance rate, kept for drill-down
sum by (job, instance) (rate(http_requests_total[5m]))

# The single busiest instance right now, as a series
topk(1, sum by (instance) (rate(http_requests_total[5m])))

# Fleet p99 latency, done correctly from histogram buckets
histogram_quantile(0.99,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Note what the latency query does: it sums the _bucket counters across instances first, then computes the quantile. Histogram buckets are additive; precomputed quantiles are not.

How to configure it

Aggregations are configured where they are computed: in queries, in alerts, and in recording rules. The recording rule file is the piece you operate as configuration:

# /etc/prometheus/rules/fleet.rules.yml
groups:
  - name: fleet-aggregates
    interval: 30s
    rules:
      # Fleet request rate per job: cheap to query, stable to alert on.
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Per-instance rate, kept for drill-down dashboards.
      - record: job_instance:http_requests_total:rate5m
        expr: sum by (job, instance) (rate(http_requests_total[5m]))

      # Fleet p99 latency from histogram buckets.
      - record: job:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99,
                sum by (job, le)
                  (rate(http_request_duration_seconds_bucket[5m])))

The naming convention is level:metric:operations: the labels that survive (job), the base metric (http_requests_total), the operations applied (rate5m). Anyone reading the recorded name can reconstruct the query. Standardise a small set of rate windows — 5m, 30m, 1h — so that every rule in the platform is comparable.

How to validate it

# 1. Syntax-check the rule file before it is loaded.
promtool check rules /etc/prometheus/rules/fleet.rules.yml
# SUCCESS: 3 rules found

# 2. Confirm the recorded series exists and matches an ad hoc query.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=job:http_requests_total:rate5m' | jq '.data.result'

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))' \
  | jq '.data.result'

# 3. Check rule group health and evaluation cost.
curl -s 'http://localhost:9090/api/v1/rules' \
  | jq '.data.groups[] | {name, interval, lastError: .rules[0].lastError}'

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_rule_group_last_duration_seconds' \
  | jq '.data.result'

Recorded and ad hoc values should match within one evaluation interval of drift. A growing prometheus_rule_group_last_duration_seconds means the aggregation is getting more expensive — usually because input cardinality grew.

How it can fail

1. The average hides the outlier. Twenty instances, one at 95 percent steal, nineteen at 2 percent: avg reads 6.65 percent and the panel is green. Symptom: users on one instance suffer while every fleet panel looks fine. Use max or a count(instances over threshold) alongside.
2. Sum over raw counters. sum(http_requests_total) adds cumulative-since-start counters. Symptom: the line only climbs, dips when an instance restarts and resets its counter, and means nothing. Apply rate() or increase() first.
3. The clause drops the label the alert needs. An alert on sum by (job) fires “api is slow” with no instance attached. Symptom: the page wakes someone up to start searching from zero. Keep the drill-down label in the by list and let Alertmanager group.
4. Averaging precomputed quantiles. avg(http_request_duration_seconds{quantile="0.99"}) is not the fleet p99; quantiles are not additive. Symptom: a latency panel that looks plausible and is mathematically meaningless. Aggregate histogram _bucket series, then histogram_quantile.
5. The by-label is high cardinality. Recording sum by (route) (...) where route has 500 values moves the cardinality; it does not shrink it. Symptom: the recording rule itself becomes the series explosion and rule evaluation time climbs.
6. Mixed rate windows. Some rules use rate1m, others rate5m, for the same metric. Symptom: two dashboards disagree about the same spike; alerts tuned against one window misbehave against the other.

How to troubleshoot it

1. Does the input exist at the expected cardinality? count(http_requests_total) and count by (job) (http_requests_total). If the input series count is wrong, the aggregation is not the problem.
2. What does the clause keep? Run the expression ad hoc in Grafana Explore and inspect the result labels. Missing instance in the output explains a dead drill-down link immediately.
3. Compare operators on the same input. Query avg, max and count of the same expression side by side. A wide gap between avg and max is the outlier-hiding failure made visible.
4. Is the recording rule evaluating? Check prometheus_rule_group_last_duration_seconds, prometheus_rule_evaluations_total, and the rules API for lastError. A rule with an error produces nothing and the downstream panels simply show “No data”.
5. Did the clause change recently? A changed by list creates a new recorded series family; the old one goes stale within five minutes. History looks like it stopped.

Security implications

Aggregation is a data-minimisation tool. A recording rule that drops instance, pod and any tenant-identifying label produces a series safe to show on a wall dashboard shared across teams. The inverse is also true: an aggregation whose by clause keeps a sensitive label (customer ID, internal hostname) leaks it to anyone with query access. The Prometheus query API is unauthenticated by default, so anyone who can reach port 9090 can run any aggregation over any label you kept. Restrict the API and review which labels recorded series retain before wiring them into shared dashboards.

Performance implications

Aggregation cost is proportional to input series count at every evaluation step. A dashboard panel running sum(rate(...[5m])) over 100,000 series on every refresh can take seconds and multiply across viewers; the same expression as a recording rule costs that once per interval. Keep recorded by label sets small and bounded. topk at query time is fine; in a recording rule it both costs the full aggregation and produces unstable output series. Long rate windows multiply cost linearly — a rate[1h] range aggregation touches twelve times the samples of rate[5m].

Production guidance

• Record fleet-level aggregates as rules with the level:metric:operations naming convention; point dashboards and alerts at the recorded series, not at ad hoc expressions.
• Always record a per-instance variant of fleet aggregates that matter operationally, so drill-down never requires re-deriving the data at query time.
• Alert on max or count over threshold for instance health, on avg or sum for fleet capacity and SLO burn. Never on avg alone for health.
• Standardise rate windows (5m, 30m, 1h) across the platform so rules, alerts and dashboards are comparable.
• Never aggregate summary quantiles; aggregate histogram buckets and compute quantiles last.

Verification

You should now be able to answer:

• Which labels survive sum by (job) (...) and which survive sum without (instance) (...)?
• Why can avg across instances hide a single failing instance, and which operators expose it instead?
• Why must counters be passed through rate() or increase() before aggregation?
• Why is averaging precomputed quantile series invalid, and what is the correct pattern with histogram buckets?
• What belongs in a recording rule and what belongs at query time?

Quiz