TraceQL Aggregations

\{ resource.service.name = "checkout-api" && status = error \} | count() > 0 returns the trace IDs of every trace whose checkout path produced at least one error span. Without the aggregator, the same query returns the error spans themselves; with by (resource.service.name), the result breaks down to “how many error traces per service”. This lesson is about the pipeline stages that turn a selector into a count.

What it is

A TraceQL aggregator is a pipeline stage that collapses a spanset (the spans that survived the prior stages in one trace) into a single value. The current set of aggregators is:

Aggregator   Purpose                                  Example
-----------  ---------------------------------------  --------------------------------
count()      Number of spans in the spanset           { ... } | count() > 3
avg(field)   Average of a numeric attribute           { ... } | avg(span:duration) > 100ms
sum(field)   Sum of a numeric attribute               { ... } | sum(span.bytes)
max(field)   Maximum value of a numeric attribute     { ... } | max(span:duration) > 2s
min(field)   Minimum value of a numeric attribute     { ... } | min(span:duration) < 5ms

The aggregator is followed by a comparator and a literal to keep or discard traces:

# Traces with more than 10 spans
{ } | count() > 10

# Traces whose average span duration exceeds 100ms
{ } | avg(span:duration) > 100ms

# Traces whose largest span exceeds 2 seconds
{ } | max(span:duration) > 2s

# Traces that total more than 1 GB of bytes processed
{ } | sum(span.bytesProcessed) > 1000000000

The by clause groups spans by an attribute before the aggregator runs. The result is per-group: one row per distinct value of the attribute.

# Count of error traces per service
{ status = error } | by(resource.service.name) | count() > 0

# Average duration per pod
{ resource.service.name = "checkout-api" } | by(resource.k8s.pod.name) | avg(span:duration) > 100ms

Without by, the aggregator produces one value per trace. With by, it produces one value per group of spans inside the trace.

Why a sysadmin cares

A selector finds spans. An aggregator turns spans into a number. That conversion is the moment TraceQL stops being a search engine and becomes a measurement. Three operational reasons aggregators matter:

1. Filtering by quantity, not by presence. “Traces with more than 50 spans” is a different question from “traces that contain this attribute”. Aggregators answer the first.
2. Breakdowns by attribute. “How many error spans per service” is the question every on-call engineer asks during a multi-service incident. The by clause answers it in one query without scanning every trace.
3. Pre-aggregated metrics. A TraceQL query with by (resource.service.name) | count() produces a per-service count. Combined with a Prometheus recording rule that scans the result periodically, this is a slow but exact span-derived metric that does not require a separate metrics-generator.

How it works

A TraceQL pipeline is a sequence of stages. Each stage takes a spanset and returns either a smaller spanset or a single value:

  { ... }              selector       keeps matching spans
  { ... } | count()    aggregator     collapses to one number per trace
  { ... } | by(...)    grouping       buckets the spanset by attribute
  { ... } | topk(10)   selection      picks the top K groups

The grouping stage runs before the aggregator. Without by, the spanset inside one trace is aggregated together. With by, the spanset is split into one subset per distinct value of the named attribute, and the aggregator runs once per subset:

  { status = error }
        |
        v
  by(resource.service.name)
        |
        +-- service=checkout-api  --> count() = 12
        +-- service=payments     --> count() = 8
        +-- service=inventory    --> count() = 3
        |
        v
  One result row per group

The comparator at the end filters groups, not traces. A query like by(resource.service.name) | count() > 5 keeps only the groups whose count exceeds 5.

Under the hood

How to configure it

Aggregators are query-side; no Tempo configuration is required to use them. The relevant configurations are the same as in lessons 01 and 02: querier.max_query_length and the query-frontend cache.

One config that affects aggregator behaviour is the search-result cache TTL on the query-frontend. A Grafana panel that groups by service benefits from a longer TTL (60 seconds); an investigation query benefits from a shorter TTL (5 seconds) so the operator sees new data.

# /etc/tempo/tempo.yaml
query_frontend:
  search:
    # Cache TraceQL search results (including aggregations)
    # for 60 seconds. Cache key is the query string plus the
    # time window.
    cache:
      enable_fifo_cache: true
      fifocache:
        size: 2048
        validity: 60s

Severity: CONFIGURATION. The query-frontend must be restarted.

How to validate it

Severity: READ-ONLY. Four checks confirm an aggregation is behaving as expected.

1. The aggregator runs and returns a number:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ resource.service.name = "checkout-api" } | count()' \
  --data-urlencode 'limit=1' | jq '.traces[0] | .traceID'

"4bf92f3577b34da6a3ce929d0e0e4736"

The trace ID is the answer; the aggregator runs inside the search and produces a single value per trace.

2. The grouping stage produces per-group counts:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ status = error } | by(resource.service.name) | count()' \
  --data-urlencode 'limit=10' | jq '.traces | length'

5

Five groups means five distinct services with error traces in the window. Compare against the ungrouped count to confirm the breakdown makes sense.

3. The aggregator comparator keeps or drops the expected traces. Add count() > 0 and confirm the same five groups return; add count() > 1000 and confirm an empty result:

# Traces with at least one error span (any number)
curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ status = error } | count() > 0' \
  --data-urlencode 'limit=10' | jq '.traces | length'

# Traces with more than 1000 error spans (likely empty)
curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ status = error } | count() > 1000' \
  --data-urlencode 'limit=10' | jq '.traces | length'

The first returns 5; the second returns 0. The comparator is filtering as expected.

4. The query-frontend cache is engaged. Repeat a query and check the cache-hit metric:

# Repeat the same query twice
curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ status = error } | count() > 0' \
  --data-urlencode 'limit=10' > /dev/null

curl -s http://tempo-query-frontend:9095/metrics | \
  grep query_frontend_search_results_cache_hits_total

A non-zero counter on the second run means the cache served the result without re-scanning blocks.

How it can fail

Six shapes appear when an aggregation returns the wrong answer:

1. The comparator is on the wrong side of zero. count() > 0 keeps traces with at least one match; count() >= 0 keeps every trace (the count is always non-negative). Symptom: the comparator looks correct but returns every trace.
2. The by attribute has unexpected cardinality. A by(span.http.url) groups by full URL, which can be unique per request. Symptom: thousands of groups, each with a count of 1; the querier runs out of memory.
3. The aggregator field is wrong. avg(span.duration) uses the bare name; the canonical intrinsic is span:duration. Symptom: the aggregator fails to parse or returns 0 for every group because the column is empty.
4. The grouping produces one group with everything. The attribute is constant (e.g. by(instrumentation.name) on a fleet that uses a single SDK). Symptom: the result has one row and the breakdown the operator expected does not appear.
5. The pipeline order is wrong. count() | by(...) is not valid syntax. The order is by(...) | count(). Symptom: the Tempo UI shows a red syntax banner.
6. The time window is too long for the cardinality. A 30-day window plus by(resource.k8s.pod.name) on a fleet with 500 pods produces 500 groups per service. Symptom: query times out; querier OOMs.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Confirm the time window contains traces. Run the selector alone ({ ... }) without the aggregator. Zero means the window is empty; the issue is retention or clock skew.
2. Confirm the aggregator parses. The Tempo UI shows a syntax error in red. Common mistakes: forgetting the parentheses (count > 0), using the bare name for an intrinsic (avg(duration) instead of avg(span:duration)), or chaining two aggregators (avg() | count()).
3. Confirm the cardinality of the by attribute. Pick a single trace, run by(...) | count() on it, and confirm the number of groups is sensible. A count that matches the number of spans means the attribute is per-span unique.
4. Confirm the comparator is on the right side of the threshold. count() > 0 means at least one. count() > 100 means more than 100. The threshold is per trace, not per time window.

Security implications

• Cardinality as a side channel. A query like by(span.user_id) | count() returns per-user counts. In a multi-tenant deployment this can leak user-level behaviour across tenant boundaries if the X-Scope-OrgID header is misrouted.
• Query cost as DoS. A panel that runs by(span.http.url) | count() against a 30-day window scans every block and produces millions of groups. One user, one panel, can saturate the querier. Cap the time window and bound the by cardinality.
• Cache poisoning. The query-frontend cache keys on the query string plus the time window. A query that returns high-cardinality results can fill the cache and evict other panels’ results.

Performance implications

The cost of an aggregation is:

• Selector cost (column reads, see lesson 02).
• Group cost. A by on a low-cardinality attribute is cheap because the engine allocates one bucket per distinct value. A by on a high-cardinality attribute is the dominant cost: millions of allocations, millions of comparator evaluations.
• Aggregator cost. count() is O(N) over the spanset; sum(), min(), max() are also O(N); avg() is O(N) with a small constant. None of them are expensive on their own.
• Comparator cost. O(G) over the groups; cheap.
• Time window cost. Multiplied by the number of blocks in the window.

The query-frontend cache turns repeat dashboard queries from O(blocks) to O(1). Set the cache TTL to the dashboard refresh interval; longer TTLs serve stale results.

Production guidance

• Always pair an aggregator with a comparator. count() alone returns a number; count() > 0 is the actionable form that keeps traces.
• Bound the by cardinality. Service-name, pod-name, and deployment-environment are sensible. URL, user-id, and request-id are not.
• Use count() > 0 to convert a selector into a trace-ID list, and use by(...) | count() to convert it into a breakdown.
• Cache aggregations at the query-frontend with a TTL matched to the dashboard refresh interval.

Verification

You should now be able to answer:

• What does count() > 0 do at the end of a TraceQL pipeline?
• How does the by clause change the result of an aggregator?
• Why is a by on span.http.url a dangerous choice in production?
• What is the cost order of count(), avg(), sum(), max(), min()?
• What is the first thing to check when an aggregation returns zero results?

Quiz