sum() and avg()

A Grafana panel sits at “instance CPU: 38%” and an on-call engineer asks the obvious question: what is the fleet doing? The operator opens the same panel scoped to job=api. The number is now 41%. Two minutes later the alerting rule fires on “fleet CPU above 90%” and the team starts paging — because someone summed per-instance gauges with no divisor, and the panel is reporting the sum of percentage points, not a fleet-wide percentage. The investigation stalls.

Two aggregators cause this pattern more often than the rest: sum() and avg(). They are simple to write, easy to misread, and the boundary between “useful” and “wrong” lives in the metric type, the label set, and whether the right thing was aggregated first.

What sum() and avg() are

Both are PromQL aggregation operators: functions that take a vector of time series and return a smaller vector. The shape of the output is controlled by an optional by or without clause that names which labels survive the reduction.

sum(http_requests_total)                       # one series, no labels
sum by (job) (http_requests_total)             # one series per job
sum by (instance) (rate(http_requests_total[5m]))  # rps per instance

avg(node_load1)                                # mean across all series
avg by (job) (node_load1)                      # mean load per job

sum() adds the sample values across the grouped series at each timestamp. avg() divides the sum by the count. Both preserve the metric type for the output vector: summing counters gives a counter, averaging gauges gives a gauge. They cannot change a counter into a gauge or a histogram into a numeric value.

Why a sysadmin cares

Three production questions get asked over and over and only these two aggregators answer them cheaply:

• “What is the fleet-wide request rate right now?” — sum(rate(...))
• “Which job is using the most memory per instance?” — avg by (job) (memory_used_bytes)
• “What is the cluster-wide error ratio?” — sum(rate(errors)) / sum(rate(requests))

All three depend on getting the boundaries right. sum() of a raw counter is nonsense (counters only make sense as rates). avg() of gauges across instances with different sample windows is wrong. sum() of averages, taken naively, produces a number unrelated to the true total. Each of these traps has cost a team a half hour of investigation in real incidents.

How it works

The mental model is a two-step pipeline:

vector of series   -->   grouping by label set   -->   reduce per group
                                                                     |
                                                          sum() / avg()
                                                                     |
                                                                     v
                                                          one series per group

The grouping is implicit if you omit by / without: Prometheus collapses every series into one. With a clause, it partitions the input vector into buckets that share the listed labels (for by) or share every label except the listed ones (for without), then runs the reducer on each bucket.

sum by (instance) (rate(http_requests_total[5m]))
                       |
                       v
+----------------------+----------------------+
| instance=web01       | instance=web02       |
+----------------------+----------------------+
| 142 req/s            | 137 req/s            |
+----------------------+----------------------+

sum() and avg() are pure functions over the current value at each timestamp. They do not look at history, do not smooth, and do not respect the time range of an inner range vector. avg() of a range vector is illegal: you must aggregate over time first (avg_over_time()), then across series.

How to configure it

sum() and avg() are query-time operators. They are not configured per se; they are written. The configuration that matters is the recording rule that precomputes them.

The canonical production pattern is to aggregate at the lowest boundary that has a label set, expose a recording rule at that level, and then aggregate the recording rule further up.

# /etc/prometheus/rules/aggregation.yml
groups:
  - name: per-instance
    interval: 30s
    rules:
      # Per-instance request rate. One series per (job, instance).
      - record: instance:http_requests:rate5m
        expr: sum by (job, instance) (rate(http_requests_total[5m]))

      # Per-instance average latency is dangerous. We compute the
      # histogram quantile per-instance instead and expose that.
      - record: instance:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, instance, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

  - name: per-cluster
    interval: 30s
    rules:
      # Fleet request rate: roll up the per-instance recording rule.
      - record: job:http_requests:rate5m
        expr: sum by (job) (instance:http_requests:rate5m)

      # Cluster total request rate: no `by` clause, collapses to one.
      - record: cluster:http_requests:rate5m
        expr: sum(instance:http_requests:rate5m)

The naming convention level:metric:operation is the Prometheus community standard. The level (instance, job, cluster) is the boundary of the aggregation; the operation is the reducer; the metric name is in the middle. Dashboards reference the recording rule, not the raw metric.

How to validate it

Validate in three places: the rule file, the running Prometheus, and the resulting series.

# 1. Static check of the rule file syntax.
promtool check rules /etc/prometheus/rules/aggregation.yml
# SUCCESS: /etc/prometheus/rules/aggregation.yml

# 2. Confirm the recording rule is loaded.
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.type=="recording") | .name'

# 3. Confirm the recording rule emitted a series.
curl -s 'http://prometheus:9090/api/v1/query?query=cluster:http_requests:rate5m' \
  | jq '.data.result[] | {metric, value}'
# {
#   "metric": {},
#   "value": [1754918400, "1287.45"]
# }

The third step is the one that matters. A rule that loaded without error but emits no series is a rule whose expression never matched the input — for example, a sum by (instance) over a metric that no longer carries an instance label. The dashboard that consumes the recording rule will be empty.

How it can fail

The high-frequency failure modes specific to sum() and avg():

1. Sum of a raw counter. sum(http_requests_total) returns the total number of requests since each instance started. The number grows without bound, drops when instances restart, and is not a rate. The dashboard line keeps climbing. The on-call engineer assumes traffic is growing.
2. Average of an already-averaged latency. avg by (instance) (rate(...)) followed by avg(...) at the cluster level averages averages. A 90th-percentile latency reported as “average latency” is meaningless; the operator looks at the dashboard, sees 80ms, and misses a 4-second tail that affects one in ten requests. The investigation finds the slow requests three hours later in customer reports.
3. The “sum of avg” trap. sum(avg by (instance) (x)) is not equal to sum(x) when the instance label set differs across calls. If the inner avg by (instance) is computed over ten instances, and the outer sum is computed after some instances have been filtered out, the total is the sum of the survivors, not the original. The dashboard under-reports whenever a host is missing.
4. Average across heterogeneous units. avg(node_load1) mixes the 1-minute load average across hosts with different core counts. A 32-core host at load 4.0 and a 4-core host at load 4.0 are in different states, but avg reports 4.0 as the answer. The team adds a “normalised load” recording rule that divides by count(node_cpu_core), and the value moves from 4.0 to 0.7. The dashboard was wrong the entire time.
5. Label drop on aggregation. sum by (job) (rate(...)) drops the instance label. The result has one row per job, not one per host. The dashboard panel that says “drill down by instance” has nothing to drill down into. The operator investigates the wrong shape.
6. Empty result silently emitted. sum(...) over an empty selector returns an empty vector. The dashboard panel renders as “No data” and is indistinguishable from a scrape failure. Alerting rules built on empty vectors are absent() conditions, not for: clauses — they fire the moment the selector has nothing to match.

How to troubleshoot it

A disciplined order, applied to any panel that “looks off”:

1. Inspect the query. What is being summed or averaged? Is the inner expression a counter, a rate, a gauge, or a histogram? Counters must be wrapped in rate() or irate() first.
2. Inspect the label set. Run the query without sum() / avg() and inspect what labels the input carries. Add by (X, Y, Z) to the aggregation and confirm that the resulting series have exactly those labels and no others.
3. Cross-check against the source. Pick a recent time window. Sum the raw counter in the database (e.g., increase(counter[5m]) over the window). Compare to sum(rate(counter[5m])) * 300. The values should match within rounding.
4. Inspect prometheus_tsdb_head_series and rule evaluation logs. A recording rule that evaluates but emits nothing has a label-mismatch problem, not a syntax problem.
5. Test the rule offline. promtool test rules runs a YAML suite of fixtures against the rule file. Add a fixture for every rule whose result is on a dashboard.

Security implications

Aggregation does not, in itself, change the attack surface. The rules to remember:

• The /api/v1/query endpoint that runs sum() and avg() is the same endpoint that runs label_values() and other cardinality-blowing functions. Treat it as an authenticated surface; the security part of this course covers the reverse proxy and basic-auth layers.
• Recording rules with weak by clauses produce series whose label sets are predictable. Do not rely on instance being present in a recording rule if the producer does not guarantee it. Production instrumentation guides the security review of the data model separately.
• Aggregation that drops labels can collapse two distinct customers into one series, which leaks cardinality across tenant boundaries. The multi-tenancy part of the course returns to this.

Performance implications

Three knobs matter:

• Cardinality. A sum by (instance) over a 10,000-host fleet produces 10,000 series. A sum(...) (no clause) produces one. The cost is in the series count, not in the reducer.
• Rule evaluation interval. Recording rules that aggregate frequently are recomputed frequently. The default 1-minute interval is fine for most rollups; tighten to 15s for SLO burn-rate panels only.
• Query cost on a dashboard. A panel that sums 100,000 series every refresh is a panel that re-evaluates the full scrape cache slice every refresh. Move the panel to a recording rule; have the dashboard read the precomputed series.

Verification

You should now be able to answer:

• Why is sum(counter) wrong, and what is the right form?
• What does by (X) do, and what labels survive?
• What is the “sum of avg” trap, and how do you avoid it?
• When is avg() a misleading number, and what should you plot alongside it?

Quiz