TraceQL Compatibility

A team builds a saved Grafana panel that pivots from error metrics into traces. The query uses { span.http.status_code >= 500 } && { span.http.method = "POST" }. The panel works for three months while the team runs Tempo 2.3. The team then upgrades to the next minor release. The panel returns zero results on Monday morning. The team investigates by running the same query in Grafana’s Explore tab and gets the expected results back. The difference is the dashboard’s automatic refresh; the new querier pool handles the new intrinsic, but the older cached results returned empty before the cache TTL had expired.

TraceQL evolution is forward-friendly: the same query runs on new and old Tempo. What breaks is when the operator or function depends on a feature the running tempo does not yet implement, or when the block format on disk does not carry the metadata the operator expects. This lesson covers the TraceQL lineage, the introspection query that proves a feature is alive, and the diagnostic that catches a silent empty result.

What it is

TraceQL is Tempo’s first-class query language for trace spans. It was announced as a preview in Tempo 2.0, promoted to beta in Tempo 2.1, and reached general availability in Tempo 2.3. The language has grown in three stages:

• Preview (Tempo 2.0 - 2.1). Tag and resource selectors, intrinsic span selectors (name, status, kind, duration), and the basic comparison operators (=, !=, >, <, >=, <=).
• Beta (Tempo 2.2). Structural operators (descendant, child, parent, sibling) and the first round of aggregators (count(), min(), max(), avg(), sum()).
• GA (Tempo 2.3). Documented public contract for the syntax, a stable set of intrinsics (name, status, kind, duration, startTime, endTime, traceDuration), comprehensive aggregators (histogram_quantile(), quantile(), rate()), and pipeline semantics for result transforms.
• Recent (Tempo 2.4 onwards). New intrinsics (rootName, rootServiceName), additional aggregators (topk, bottomk, compare, coalesce), and conditional aggregations.

Each stage is backward-compatible at the syntax level. A query that uses only operators from the GA stage is valid in every later release. A query that uses an operator from a later stage fails predictably in an older Tempo: the binary returns 400 unknown operator or, when the operator parses but is not implemented, returns zero results.

Why a sysadmin cares

Three production scenarios apply:

1. Saved dashboard silently returns empty. A query that references an intrinsic available only in Tempo 2.4+ returns zero results in a 2.3 cluster. The Grafana panel is green because the response is 200 OK and zero rows.
2. Alert silently suppressed. An alert that uses TraceQL to count error spans returns the count as zero because the operator is not implemented. The on-call never knows.
3. Block format not carrying the field. A query that references an attribute not present in the on-disk block header is well-formed but matches nothing. A cluster that has not been re-compacted by the latest compactor cannot match the new query against the old block metadata.

The cost of a TraceQL evolution surprise is not the error message. The cost is the dashboard that says nothing on a day the team is watching for something.

How it works

TraceQL runs through three gates before the answer reaches the Grafana panel:

   Grafana panel / Explore
        |
        v
   +-------------------+
   | Query parse       |  parse the TraceQL expression;
   |                   |  reject syntax errors with 400
   +-------------------+
        |
        v
   +-------------------+   the query-frontend plans the
   | Query plan        |   query: which intrinsics are
   |                   |   referenced, which aggregators,
   +-------------------+   which structural operators
        |
        v
   +-------------------+   the querier distributes the
   | Fan-out across    |   subqueries to the block
   | blocks            |   locations the compactor has
   +-------------------+   indexed
        |
        v
   +-------------------+   merge results at the
   | Aggregate /       |   query-frontend, applying
   | transform         |   the pipeline steps
   +-------------------+
        |
        v
   Grafana panel

The three gates are deterministic. A query that fails at parse returns 400; a query that fails at plan returns zero; a query that succeeds at plan but matches no data also returns zero. The three outcomes produce three different operational signals.

Under the hood

How to configure it

TraceQL itself is not configured. The configuration that affects TraceQL behaviour lives in the query-frontend and in the compactor’s search-tag extraction:

query_frontend:
  search:
    max_concurrent_queries: 200
  results_cache:
    cache:
      embedded_cache:
        max_size_items: 1024
        ttl: 1h

compactor:
  compaction:
    # Tags that the compactor indexes on the merged block.
    # These are the only attributes the querier can
    # filter on without doing a full block scan.
    search_relevant_tag_values:
      - resource.service.name
      - span.http.method
      - span.http.status_code
      - span.http.url

Three details to call out:

• compactor.compaction.search_relevant_tag_values is the list the compactor indexes when it rewrites a block. A TraceQL query that references an attribute outside this list works on the new blocks but does a full block scan on the old blocks. The performance regression looks like a query-latency panel that jumped up after an upgrade.
• query_frontend.results_cache.cache.embedded_cache.max_size_items is the bound on how many sub-results the cache remembers. A cache that is too small thrashes; a cache that is too large consumes memory the querier pool needs.
• search.max_concurrent_queries bounds the fan-out. A TraceQL query with count() over a large time range fans out across every matched block; the bound protects the querier pool.

How to validate it

Severity: READ-ONLY.

1. Confirm the running Tempo’s TraceQL feature set. The tempo_querier_traceql_queries_total metric exposes the versions the querier has handled:

curl -s http://tempo-querier:3200/metrics \
  | grep '^tempo_querier_traceql_queries_total'
# tempo_querier_traceql_queries_total{type="intrinsic_name"} 1842
# tempo_querier_traceql_queries_total{type="intrinsic_rootName"} 92

A new intrinsic counter that is absent is the first signal that the running tempo does not yet implement it.

2. Confirm the saved Grafana panel still returns results after the upgrade. The shape of the panel’s table is the proof:

curl -sG http://tempo-querier:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" }' \
  --data-urlencode 'limit=5' \
  | jq '.traces | length'
# 5

3. Confirm an intrinsic available only in 2.4+ returns results on the running tempo:

curl -sG http://tempo-querier:3200/api/search \
  --data-urlencode 'q={ rootName = "GET /api/cart" }' \
  --data-urlencode 'limit=5' \
  | jq '.traces | length'
# 5 if rootName is recognised, 0 if not.

4. Confirm a known trace matches the TraceQL filter. A non-zero result is the proof the join between the block metadata and the trace data works:

TRACE=8d3b4e3b3a1c4f5a92a3f3b1d5e0a4f9
curl -sG http://tempo-querier:3200/api/search \
  --data-urlencode "q={ traceID = \"$TRACE\" && span.http.status_code >= 500 }" \
  | jq '.traces | length'
# 1

5. Confirm the compactor is indexing the attribute the dashboard references. The block-level meta.json carries search_relevant_tag_values:

# Substitute your own value before running: the tenant and block ID,
# from `tempo-cli list blocks` or an `aws s3 ls` of the bucket.
BLOCK_PREFIX=single-tenant/0f8fad5b-d9cb-469f-a165-70867728950e

aws s3 cp "s3://tempo-traces-prod/$BLOCK_PREFIX/meta.json" - \
  | jq '.search_relevant_tag_values'
# [
#   "resource.service.name",
#   "span.http.method",
#   "span.http.status_code"
# ]

How it can fail

Five shapes appear in TraceQL evolution:

1. Intrinsic not implemented. A query that uses rootName on a 2.3 tempo returns zero results. The parser succeeds, the plan succeeds, the fan-out is empty. Symptom: the panel is green and blank.
2. Aggregator not in the running version. histogram_quantile() joins per-span data into a quantile result. A cluster that has not been recompiled returns the count instead of the quantile. Symptom: the panel shows a flat line for the quantile and a different shape for the count.
3. Block format not indexed. A query that references span.db.statement on a bucket that has not been recompacted with search_relevant_tag_values does a full scan. Symptom: query latency panel jumps by a factor of five to ten.
4. Cached empty result. A saved dashboard caches an empty response. The cache TTL passes before the dashboard refresh. Symptom: the dashboard shows empty results for up to the cache TTL even after the underlying query has been fixed.
5. Aggregator on a structural operator mismatch. Some aggregators require a top-level selector to be of a specific shape; a query that mixes a structural operator with an incompatible aggregator returns zero rows. Symptom: a comparison that worked in the GA version returns empty in a later version.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Is the intrinsic in the running version? Compare the query intrinsic list against the Tempo source for the running binary. The intrinsic support table lives in tempo/modules/traceql/.
2. Is the compactor indexing the attribute? compactor.compaction.search_relevant_tag_values. The metadata file of a recent block carries the indexed list.
3. Is the cache TTL expired? A query that returned empty ten minutes ago might be the cached version of a fix that was applied two minutes ago.
4. Is the fan-out empty by design? A new TraceQL filter on a service that has no traces in the queried window returns empty for legitimate reasons. Confirm with a wider time range.
5. Is the query plan valid? tempo_querier_traceql_queries_total series labels record the query types the querier has handled. A label absent from the metric is unimplemented.

Security implications

TraceQL is a read-only query language. Its security surface is the read path:

• Query complexity. A TraceQL query that references complex structural operators fans out across many blocks per second. The query is a denial-of-service vector if the query-frontend bound (max_concurrent_queries) is not enforced. Tenants should be rate-limited per-second on query fan-out.
• Trace-level PII. A query that selects span.http.url = "/api/users/123/profile" returns a list of traces for one specific user. The query is logged in the query-frontend log and should be redactable by the existing request log redaction policy.
• Per-tenant access. Tempo’s per-tenant quota configuration controls max bytes per query and max queries per second. TraceQL did not weaken or change the access controls but the larger intrinsic surface increases the effective surface for legitimate-looking queries that exhaust a quota.

Performance implications

The cost of a TraceQL query depends on the operator set:

• Pure structural. A pure structural operator (descendant, sibling) does a tree-walk per span set returned. The cost scales with the number of spans in the set.
• Aggregators. count() is a constant-cost reducer. avg() and sum() are linear. histogram_quantile() does a quantile join across the time range.
• Indexed attributes. A filter on resource.service.name is index-bound; a filter on a non-indexed attribute is a full block scan.
• Pipeline length. A pipeline with many stages applies each stage in turn. The intermediate result size dominates the cost more than the final result size.

Production guidance

• Pin compactor.compaction.search_relevant_tag_values to the list your dashboards reference. Recompile the cluster with the new list before a TraceQL dashboard assumes the new attribute.
• Audit saved Grafana panels for TraceQL intrinsics before upgrading Tempo. A panel that uses a 2.4+ intrinsic returns empty after a downgrade, not an upgrade.
• Bound query_frontend.search.max_concurrent_queries so a TraceQL regex across weeks of trace data cannot pin a querier pool.
• Treat the tempo_querier_traceql_queries_total metric as the source of truth for which intrinsics the running querier implements. The counter labels change between versions.

Verification

You should now be able to answer:

• In which Tempo version did TraceQL reach general availability?
• Which intrinsic selectors were added in the 2.4+ releases?
• What is the relationship between the compactor’s search_relevant_tag_values list and a TraceQL query that references an attribute not in the list?
• What is the first diagnostic that confirms a TraceQL intrinsic is implemented in the running Tempo?
• What panel-level alert catches the silent-empty regression?

Quiz