Poor Aggregation

A panel evaluates http_requests_total. No aggregation. The metric carries eight hundred thousand series. The panel returns the first one hundred series that match the matchers, and Grafana shows a stack of one-hundred-line charts. The chart has no meaning. The query took fourteen seconds to return one hundred series out of eight hundred thousand.

The panel was authored without aggregation because the author expected the metric to be small. It was small in the staging environment. In production, with eight hundred services and four labels per service, it is not small. The panel expression has not been updated.

This lesson is about the slow-query shape where the cost is paid in the pipeline phase, not the index or the scan. The fix is to aggregate at the layer where the data is naturally aggregated: the recording rule. The panel reads from the rule. The pipeline cost is paid once per interval.

What a poor-aggregation query is

A poor-aggregation query is one that aggregates at the panel instead of at the recording rule, repeating work the rule should already have done. The pipeline cost is proportional to the number of points produced per step and the number of operator applications per point.

  cost  =  series_scanned  x  pipeline_ops  x  work_per_step

       =  N                x  function_calls  x  points

The lowest cost layer is the recording rule. The exporter is not a place to aggregate (the exporter becomes stateful; crashes lose data). The panel is the worst place to aggregate because the cost is paid by every consumer independently.

The classic shape is a panel that sums or averages a high-cardinality metric at refresh time:

# Bad: aggregates at the panel for every refresh.
sum by (service) (rate(http_requests_total[5m]))

vs.

# Good: reads from a recording rule.
sla:http_requests:rate5m_by_service

The expressions look similar. The cost is different by a factor of the number of panels and alerts that read the metric. Every consumer of the original expression pays the full pipeline cost; every consumer of the rule pays a series lookup.

Why a sysadmin cares

A panel that aggregates at refresh time is the slow-query shape that hides the longest. The first time the panel is opened, the cost is borne by a single user. The cost is repeated on every refresh. When the dashboard is added to the wall-display behind the engineering team, the cost is borne continuously. When the dashboard is referenced from an external team’s monitor, the cost compounds.

The user-visible symptom is a panel that takes seconds to load. The platform-visible symptom is query slots occupied by what should have been a series lookup. The root cause is a panel expression that was not refactored when the metric scaled.

How to detect a panel that aggregates wrong

The same engine metrics apply. The diagnostic that distinguishes poor-aggregation from high-cardinality is the ratio of evaluation cost to series count.

# READ-ONLY. Evaluation cost per series, top ten.
# A high ratio indicates a per-series-expensive pipeline.
promql='topk(10,
  sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m]))
  /
  clamp_min(count by (query) (rate(prometheus_engine_query_duration_seconds_count[5m])), 1)
)'
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query \
  | jq '.data.result[] | {query: .metric.query, cost_per_eval: .value[1]}'

{
  "query": "sum by (service) (rate(http_requests_total[5m]))",
  "cost_per_eval": "0.84"
}

Eight hundred milliseconds per evaluation of the query is typical for a panel that aggregates a high-cardinality metric at refresh time. The metric is the ratio of total evaluation time to evaluation count.

The second diagnostic is to inspect the panel expression itself. Open the panel, expand the query inspector, and read the expression. An expression that aggregates a high-cardinality metric at the panel is a candidate for a rule.

How to fix it

The fix is structural. Three layers, in order of preference.

Layer 1. The recording rule. Pre-aggregate the expensive expression into a rule. The rule evaluates once per interval. The result is cached for every consumer.

# /etc/prometheus/rules/sla.yml
groups:
  - name: sla.http_requests
    interval: 30s
    rules:
      # Pre-aggregate the per-service rate over a 5-minute window.
      # The panel reads from this rule.
      - record: sla:http_requests:rate5m_by_service
        expr: |
          sum by (service, status) (
            rate(http_requests_total[5m])
          )

      # Pre-aggregate over a 1-hour window for wider panels.
      - record: sla:http_requests:rate1h_by_service
        expr: |
          sum by (service, status) (
            rate(http_requests_total[1h])
          )

The panel expression becomes:

sum by (status) (sla:http_requests:rate5m_by_service)

The rule pre-aggregated by service and status. The panel sums away service to display a single stacked-bar chart of status. The pipeline cost at the panel is the aggregation of roughly one hundred series (one per service) over the rule output. The cost at the rule is the aggregation of eight hundred thousand series over the original expression, but it pays that cost once per interval.

Layer 2. The aggregator exporter. When the metric itself is too high-cardinality for Prometheus to aggregate at all, push the aggregation upstream of the platform. A common pattern is an OpenTelemetry Collector processor that aggregates metrics before they leave the agent.

# /etc/otelcol/config.yaml -- relevant fragment.
processors:
  metric_transform:
    transforms:
      - include: http_requests_total
        action: update
        operations:
          - action: aggregate_labels
            label_set: [service, status]
            aggregation_type: sum

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [metric_transform, batch]
      exporters: [prometheus]

The Collector sums http_requests_total by service and status before the series reach the platform. The platform sees one hundred series, not eight hundred thousand.

Layer 3. The alert. An alert that aggregates at evaluation time is also a poor-aggregation shape. The alert is evaluated on every tick; the aggregation cost is paid on every tick. Replace the alert expression with a rule reference.

# /etc/prometheus/rules/alerts.yml
groups:
  - name: sla.alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum by (service) (rate(sla:http_requests:rate5m_by_service{status=~"5.."}[5m])) > 0.05
        for: 5m
        labels:
          severity: page

The alert reads from the rule. The rule does the aggregation. The alert pays a series lookup.

How to validate it

Three steps.

Step 1. Confirm the rule is loaded and healthy.

# READ-ONLY. Lists the recording rules in the sla.http_requests group.
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[] | select(.name == "sla.http_requests")'

{
  "name": "sla.http_requests",
  "file": "/etc/prometheus/rules/sla.yml",
  "interval": "30s",
  "rules": [
    {
      "name": "sla:http_requests:rate5m_by_service",
      "query": "sum by (service, status) (rate(http_requests_total[5m]))",
      "health": "ok",
      "lastEvaluation": "2026-08-13T03:00:00Z",
      "lastEvaluationDuration": 0.0114
    }
  ]
}

health: ok, an evaluation within the interval, and a duration in tens of milliseconds indicate the rule is healthy.

Step 2. Confirm the panel reads from the rule.

Open the panel in Grafana, expand the query inspector, and read the panel expression. The expression should reference the rule name (sla:http_requests:rate5m_by_service) and not the original metric (http_requests_total).

Step 3. Confirm the result matches.

# READ-ONLY. Compare the result of the panel expression
# against the original expression at the same instant.
rule='sum by (status) (sla:http_requests:rate5m_by_service)'
expr='sum by (status) (rate(http_requests_total[5m]))'

curl -s --data-urlencode "query=${rule}" http://prometheus:9090/api/v1/query \
  | jq '.data.result' > /tmp/rule.json
curl -s --data-urlencode "query=${expr}" http://prometheus:9090/api/v1/query \
  | jq '.data.result' > /tmp/orig.json

diff /tmp/rule.json /tmp/orig.json && echo "match" || echo "drift"

A drift indicates the rule has not aggregated at the same level as the panel. The fix is to update the rule; do not silence the diff.

How it can fail

Five failure shapes.

1. Rule aggregates by the wrong labels. The rule sums by cluster, service, status; the panel sums by service, status. The rule output has more series than the panel needs, but it has all the panel needs. Drift between the rule output and the panel output is the result.
2. Rule interval is too long for the panel’s freshness. A rule with interval: 5m serves a debugging panel that needs minute-to-minute freshness. The user sees data that is five minutes stale.
3. Rule output cardinality is higher than the original. The rule uses histogram_quantile on a metric that does not have a histogram bucket layout. The rule output contains more series than the original; the rule is a net loss.
4. Panel and alert use different aggregation levels. The panel uses the rule; the alert uses the original. The alert pays the full cost on every tick.
5. Two rules do the same aggregation with different naming. A team writes sla:http_requests:rate5m_by_service; another team writes api:http:rate_by_svc. The dashboard library grows by one rule per team per metric per dashboard. The cost of the rule library grows proportionally.

How to troubleshoot it

1. Find the offender. topk(10, sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m]))). Note the queries whose expressions aggregate a high-cardinality metric.
2. Count the consumers. For each offending expression, count the number of panels and alerts that reference it. A high consumer count is a high multiplier on the cost.
3. Write the rule. Apply the rule pattern from this lesson. The rule name follows level:metric:operations.
4. Validate the rule. Diff the rule output against the panel expression at the same instant.
5. Repoint the consumers. Update every panel and alert to read from the rule. The cost falls on the next evaluation.

Security implications

A recording rule is a privileged query; the same caveats as the large-range lesson apply. The rule file lives on disk and is evaluated by the Prometheus process. An attacker who can write the rule directory can define an arbitrarily expensive expression that runs every thirty seconds. Restrict write access to the rule directory with file permissions and run Prometheus as a non-root user.

A rule that aggregates away a privacy-sensitive label (such as user_id) does not remove the label from the original metric. The original metric still carries the label in the TSDB. The rule produces a derived series without the label; the original still exists. Drop the label at relabel-config if the platform does not need the raw series.

Performance implications

• CPU. Rule evaluation burns CPU once per interval per rule. A rule with an expensive expression and a 30 s interval burns 2 % of a core per minute per rule.
• Memory. The rule result set must fit in the evaluation buffer. A rule that returns millions of series out-of-memory kills the evaluation.
• Disk. Rules are stored as new series. A rule with the same cardinality as its source doubles the TSDB footprint.

Production guidance

• Default every panel expression to read from a recording rule. Panels that aggregate raw series are exceptions reserved for low-cardinality metrics.
• Centralise the rule library in a single repository with a single review gate. The naming convention level:metric:operations makes ownership visible.
• Audit dashboards for panel expressions that aggregate high-cardinality metrics. A panel that runs a per-service rate expression against a metric with more than ten thousand series is a candidate for a rule.
• Alert on rule evaluation failures. increase(prometheus_rule_evaluation_failures_total[5m]) > 0 fires when a rule failed to evaluate. A failed rule silently serves stale data.
• Back up the rule files. A lost rule is harder to rebuild than a lost dashboard.

Verification

You should now be able to answer:

• What four layers can aggregate a metric, and which is the right one in production?
• How does the cost of a poor-aggregation query grow with the number of consumers?
• Why does a panel that reads from a recording rule cost less than one that aggregates the raw metric?
• What is the naming convention for a recording rule, and what do the three parts mean?

Quiz