Over-Aggregation Risks

An alert fires on “fleet-wide error rate above 2%.” The on-call engineer opens the panel. The number is 2.4%. The engineer pages the team. Forty minutes later the postmortem identifies the culprit: one host, returning 100% errors on /checkout. The fleet average was 2.4% because eleven healthy hosts were diluting the broken one. The alert was right; the dashboard was useless. The team could not see which host to investigate.

The panel had aggregated the fleet-wide error rate by summing across instances and dividing by the total. The host identity was gone. The on-call engineer started the investigation with no hint about where to look. The investigation cost an hour.

Over-aggregation is the act of dropping labels before the operator needs them. The labels the operator needs are the labels the dashboard cannot recover.

What over-aggregation is

Four failure shapes, all caused by aggregating one step too early or one level too high:

1. Hiding outliers. An aggregation collapses a distribution to a single number. The number is “fine” while one or two members of the distribution are not. avg() of a long-tailed distribution is the canonical example.
2. Losing attribution. An aggregation drops the label that identifies which instance, service, or customer is the problem. sum by (job) (rate(metric[5m])) keeps job but drops instance. The dashboard knows which service is slow; it cannot say which host.
3. Double-counting. Re-aggregating an already-aggregated series overcounts. sum(metric) plus sum(other_metric) where other_metric was already a sum produces a number that counts the same observation twice.
4. Aggregating in the wrong place. Computing the same aggregation inside every dashboard panel, instead of once in a recording rule. The aggregation re-runs on every refresh against the full scrape cache slice.

The first two are about labels: the right labels must survive the aggregation. The last two are about placement: the aggregation should happen once, at the right boundary, in a recording rule.

Why a sysadmin cares

Three operational pains over-aggregation causes:

• Slow investigations. The dashboard shows the right number for the wrong question. The operator knows the symptom but not the cause. Ten minutes turn into an hour.
• Slow dashboards. A panel that re-aggregates 50,000 series every refresh spends most of its wall time in the aggregation pass. Move the aggregation to a recording rule.
• Re-architecture under pressure. A team that aggregated away the instance label during a dashboard redesign cannot answer per-host questions. The fix is to rebuild the recording rule hierarchy, which is a multi-day project that nobody owns.

Each of these has a known shape and a known fix.

How it works

The mental model is a label-survival tree. Every aggregation drops some labels and keeps others. The operator’s drill-down chain is the path from the root of the tree to the leaf.

raw metric (per scrape, every label)
   |
   +-- sum by (job, instance)         -- keeps: job, instance
   |      |
   |      +-- sum without (instance)  -- keeps: job (+ everything else)
   |             |
   |             +-- sum without (job, instance)  -- one series
   |
   +-- max by (job, instance)         -- keeps: job, instance
   |
   +-- topk(10, sum by (instance))    -- keeps: instance (subset)

Each level of the tree drops labels. A level that drops a label the next level needs is a level that has been aggregated too far. The fix is to expose the lower level as a recording rule so the next level can re-aggregate from it without re-scanning the raw metric.

The four failure shapes map to the tree:

1. Hiding outliers:  avg() across instances picks the average
                      over the worst host.

2. Losing attribution: sum by (job) drops instance; the next
                       level cannot recover which instance
                       contributed.

3. Double-counting:   sum(per_host_sum) where per_host_sum is
                      itself a sum of raw counters; the total
                      counts the same observation multiple times
                      if the hierarchy is not strictly nested.

4. Wrong place:       every panel re-aggregates the raw metric;
                      a recording rule would cache the result.

How to configure it

The recording-rule hierarchy. The pattern is to aggregate at the lowest boundary that has a stable label set, expose that level as a recording rule, and aggregate the recording rule further up.

# /etc/prometheus/rules/hierarchy.yml
groups:
  - name: per-instance
    interval: 30s
    rules:
      # The bottom of the hierarchy. Per-instance, per-job.
      # Drill-down level: every label the operator may need.
      - record: instance:http_requests:rate5m
        expr: sum by (job, instance) (rate(http_requests_total[5m]))

      - record: instance:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, instance, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

  - name: per-job
    interval: 30s
    rules:
      # Mid-level. Roll up the per-instance rule. Preserve every
      # label the operator may want to slice by except instance.
      - record: job:http_requests:rate5m
        expr: sum without (instance) (instance:http_requests:rate5m)

      - record: job:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

  - name: cluster
    interval: 30s
    rules:
      # Top level. Fleet total. One series.
      - record: cluster:http_requests:rate5m
        expr: sum without (instance) (job:http_requests:rate5m)

      - record: cluster:http_request_duration:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
          )

The hierarchy is bottom-up. Each level reads from the level below. The naming convention (level:metric:operation) makes the boundary explicit. The operator who wants to drill from “cluster:http_requests:rate5m” to “instance:http_requests:rate5m” does so by changing the metric name, not by re-aggregating the raw metric.

The cardinality budget

Each level of the hierarchy is a separate set of series in the TSDB. A fleet of 50 hosts × 5 jobs × 3 routes is 750 series at the per-instance level. The per-job level is 5 × 5 × 3 = 75 series. The cluster level is 1 series. The total is the sum.

A recording rule that emits a level of the hierarchy without being consumed by a higher level is a recording rule that spends cardinality without producing value. The performance part of the course returns to cardinality budgets.

How to validate it

# 1. Static check.
promtool check rules /etc/prometheus/rules/hierarchy.yml
# SUCCESS: /etc/prometheus/rules/hierarchy.yml

# 2. Confirm the per-instance rule has every label.
curl -s 'http://prometheus:9090/api/v1/query?query=instance:http_requests:rate5m' \
  | jq -r '.data.result[0].metric | keys | join(",")'
# instance,job

# 3. Confirm the per-job rule dropped instance.
curl -s 'http://prometheus:9090/api/v1/query?query=job:http_requests:rate5m' \
  | jq -r '.data.result[0].metric | keys | join(",")'
# job

# 4. Confirm the cluster rule collapsed to one series.
curl -s 'http://prometheus:9090/api/v1/query?query=cluster:http_requests:rate5m' \
  | jq '.data.result | length'
# 1

# 5. Confirm the cluster total equals the sum of per-job totals.
JOB_TOTAL=$(curl -s 'http://prometheus:9090/api/v1/query?query=job:http_requests:rate5m' \
  | jq '[.data.result[].value[1] | tonumber] | add')
CLUSTER_TOTAL=$(curl -s 'http://prometheus:9090/api/v1/query?query=cluster:http_requests:rate5m' \
  | jq '.data.result[0].value[1] | tonumber')
echo "jobs: $JOB_TOTAL, cluster: $CLUSTER_TOTAL"
# jobs: 1287.4, cluster: 1287.4

The fifth step is the double-counting check. If the cluster total is not the sum of the per-job totals, the hierarchy is wrong — the per-job rule was computed independently from the raw metric, not from the per-host rule, and the two paths disagree because the raw metric changed between evaluations.

How it can fail

The high-frequency over-aggregation failure modes:

1. Hiding the outlier with avg(). A fleet-wide avg() over a long-tailed distribution returns “fine” while one member of the distribution is broken. The SLO claim built on the average is unsupportable. Alert on max by (instance) for the worst-offender signal; pair the average with min and max to expose the shape.
2. Losing attribution with sum by (job). A per-job dashboard that drops instance cannot answer “which host is the problem.” The on-call engineer knows the service is slow but cannot SSH to the right machine. Always include instance in the by clause of per-host recording rules.
3. Double-counting with nested sums. sum(sum by (instance) (rate(metric[5m]))) is correct because the inner sum is a rate. sum by (instance) (sum by (job, instance) (rate(metric[5m]))) is correct because the inner sum preserved instance. sum(sum(metric)) is wrong because the inner sum collapsed across instances and the outer sum counted each instance’s total once. The hierarchy must be strictly nested in the dimensions it preserves.
4. Aggregating inside every dashboard panel. A fleet-wide panel that re-aggregates the raw metric on every refresh is a panel that re-evaluates the full scrape cache slice every refresh. Move the aggregation to a recording rule.
5. Dropping the label that the alert needs. An alert on sum by (job) (rate(errors[5m])) > 10 fires for whichever job exceeds the threshold. The alert labels include job but not instance. The on-call engineer pages the job but not the host. Include instance in the by clause so the alert includes the host identity.
6. Aggregating before rate(). avg(sum(metric)) averages cumulative counters, not rates. The number grows over time and is not meaningful. Wrap each metric in rate() first, then aggregate.

How to troubleshoot it

When an investigation stalls because the dashboard does not have the labels it needs:

1. Inspect the label set of the dashboard’s source series. If the source series has instance and the dashboard does not, the by clause dropped it. Add instance to the by clause (or remove the by clause and rely on a separate per-host panel).
2. Confirm the recording-rule hierarchy is strictly nested. Per-host rule should be a sum of the raw metric. Per-job rule should be a sum of the per-host rule. Cluster rule should be a sum of the per-job rule. Any deviation is a point where the hierarchy can disagree with itself.
3. Cross-check cluster total against sum of per-job totals. If they differ, one of the levels is computed independently from the raw metric and the hierarchy is not strictly nested. Fix the recording rule to read from the level below.
4. Inspect the dashboard query’s evaluation cost. A panel that re-aggregates 100,000 series every refresh shows up in /api/v1/query?query=prometheus_engine_query_duration_seconds as a slow query. Move the aggregation to a recording rule.
5. For alerts, ensure the by clause includes the labels the alert needs to identify the offender. A fleet-wide alert that drops instance cannot page the right host.

Security implications

• A recording rule that drops instance keeps job (or equivalent). If job is not sensitive, the rule is fine. If instance is sensitive (it identifies a single host by name), dropping it is a security improvement. If instance is needed for incident response, dropping it is a security cost. The choice is operational, not security-driven.
• Double-counting in a recording rule does not leak data; it produces wrong numbers. The security implication is that the wrong numbers can be cited in compliance reports.
• Aggregating inside dashboard panels exposes the raw metric query to every Grafana viewer. A recording rule collapses the query to a single metric name, which is easier to audit.

Performance implications

• A dashboard that aggregates the raw metric every refresh spends the full aggregation cost on every refresh. A recording rule pays the cost once per evaluation interval (default 1 minute, configurable per group).
• A per-host recording rule on a 10,000-host fleet is 10,000 series. The per-job rollup is 10 series (one per job). The cluster rollup is 1. The TSDB cost is dominated by the per-host level.
• Double-counting in a recording-rule hierarchy inflates the apparent cost of an aggregation without producing any additional information. The performance part of the course returns to cardinality budgets.

Verification

You should now be able to answer:

• What are the four failure shapes caused by over-aggregation?
• Why is avg() over a long-tailed distribution misleading?
• What does “losing attribution” mean, and how do you avoid it?
• Where should aggregation happen — in the dashboard query, or in a recording rule?

Quiz