TraceQL Introduction

At 03:14 a checkout latency alert pages the on-call engineer. The histogram says POST /checkout is at 1.4 s p99. The engineer opens Grafana Explore, types \{ resource.service.name = "checkout-api" && status = error \} into the Tempo query bar, and three trace IDs appear. They click the slowest one. The trace shows a 4.1 s child span called payment-charge against payment-svc. The root cause is in front of them ninety seconds after the page.

The query that produced those three trace IDs is TraceQL. This lesson is about the language: what it is, why it exists, how it traverses a trace tree, and how to write the first query.

What it is

TraceQL is the Tempo query language for selecting traces. It borrows the syntax shape of PromQL and LogQL: an expression in curly braces that the engine evaluates against every trace in the time window. The output is a list of trace IDs, or a stream of spans, or a single value, depending on the pipeline shape.

A TraceQL expression is built from three pieces:

TraceQL = selectors { ... }
        | aggregators count(), avg(), max(), min(), sum()
        | grouping by(...) and pipeline operators

A selector is a filter over one span. The engine walks every trace in the time window, applies the selector to every span, and keeps the spans that match. An aggregator collapses a set of matching spans into a single value (count, average, sum, min, max). A pipeline chains selectors and aggregators with |.

Two query shapes cover almost every production investigation:

  TraceQL
       |
       +-- Search:   find the trace IDs that match a filter
       |     { ... }                   --> list of trace IDs
       |     { ... } | count() > 0     --> trace IDs with N+ matches
       |
       +-- Aggregate: turn a spanset into a number or a series
             { ... } | count() by (resource.service.name)
             { ... } | avg(span:duration) by (resource.k8s.pod.name)

Search answers “which traces contain a span matching this filter”. Aggregate answers “how many / how long, broken down by what”.

Why a sysadmin cares

Tempo stores every trace it receives, but only the trace IDs the operator already knows are useful. A trace ID from a log line, from an exemplar on a histogram, or from a Slack thread is the fast path. Every other investigation needs a filter to find the trace ID first.

TraceQL is that filter. Three operational pains it removes:

1. The wall of traces. A naive Tempo UI showing the last 1,000 traces is unusable for an incident with hundreds of services. A TraceQL filter cuts the list to the ten that matter.
2. The grep-by-timestamp hunt. Without TraceQL, finding the trace of one slow request means reading the application log, copying the trace ID, pasting it into the trace backend, and hoping it is still in retention. A TraceQL query against the right attribute (HTTP method, status code, user route) finds the trace in one step.
3. The metric with no follow-through. A histogram says latency is up. The exemplar on the slow bucket gives one trace. The engineer needs similar traces to see the pattern. A TraceQL query on trace:duration and status = error returns dozens.

How it works

The mental model is a pipeline. Each stage takes a set of spans and produces a smaller set of spans (or collapses it to a single value):

  All spans in time window
            |
            v
  Selector:  { resource.service.name = "checkout-api" }
            |   keeps spans whose service is checkout-api
            v
  Selector:  { status = error }
            |   keeps the error spans only
            v
  Aggregate: | count() > 0
            |   keeps traces with one or more matching spans
            v
  Result:    list of trace IDs

The engine evaluates the pipeline against one trace at a time. A trace survives the pipeline if the final stage leaves at least one span in the spanset; otherwise the trace is dropped from the result.

Three scopes name where a field lives:

• span. — attributes attached to the span itself (HTTP method, URL, status code, the span name).
• resource. — attributes attached to the resource that emitted the span (service name, service version, Kubernetes pod name, cloud region).
• event. — attributes on a span event (an exception message, a log line attached to the span).

A selector without a scope (for example status = error or name = "POST /checkout") is shorthand for the corresponding intrinsic (span:status and span:name respectively). The scope-prefixed form is canonical and the form the engine indexes.

Under the hood

How to configure it

The Tempo configuration that affects TraceQL lives under three keys: querier, query_frontend, and the storage backend. The minimal config for a TraceQL search against local data:

# /etc/tempo/tempo.yaml
querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095
    parallelism: 4
  # Cap the time window per search. Without this, a Grafana
  # panel can scan the full retention period.
  max_query_length: 168h        # 7 days

query_frontend:
  search:
    max_concurrent_queries: 20
  # Subqueries split across queriers in parallel.
  split_queries_by_interval: 15m
  max_parallelism: 4

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1

Severity: CONFIGURATION. The querier and the query-frontend must be restarted to apply.

The Grafana side is a Tempo data source with internal linking enabled. Without the link, the Grafana “Query with Tempo” action on a trace ID does not resolve:

# /etc/grafana/provisioning/datasources/tempo.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo.internal:3200
    jsonData:
      httpMethod: POST
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: -1h
        spanEndTimeShift: 1h
      # Required for the trace-id link to work.
      serviceMap:
        datasourceUid: prometheus

How to validate it

Severity: READ-ONLY. Three checks confirm TraceQL is live.

1. The Tempo querier is up and the search endpoint responds:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ resource.service.name = "checkout-api" }' \
  --data-urlencode 'limit=2' | jq '.traces | length'

2

A non-zero count with a known service name confirms the search endpoint is accepting queries and returning matches. Zero is not proof of a bug — it can mean the time window has no traces.

2. A known trace ID returns the full tree:

TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" \
  | jq '.resourceSpans | length'

1

The top-level resourceSpans array has one entry per service that contributed spans to the trace. A single service means a broken context propagation; see the troubleshooting lesson in this module.

3. The query-frontend is in the path (only relevant for microservices deployments):

curl -s http://tempo-query-frontend:9095/status \
  | jq '.queriers'

["10.0.1.12:3200","10.0.1.13:3200","10.0.1.14:3200"]

A non-empty querier list means the query-frontend has registered the queriers and is dispatching work to them.

How it can fail

Six shapes appear repeatedly when TraceQL returns the wrong answer:

1. The query that returns every trace. The selector is empty or trivially true ({ }, { span:duration > 0ns }). Symptom: the result set has thousands of trace IDs; the Grafana panel times out rendering.
2. The query that returns nothing. The attribute name is wrong, the scope is wrong (.service.name is a log label, resource.service.name is the TraceQL field), or the time window has no data. Symptom: empty result set; the engineer concludes traces are missing when they are present but the query is wrong.
3. The query that returns the right trace IDs but the wrong spans. The selector matches a parent span, not the slow child. Symptom: every returned trace looks fine on inspection because the engineer is looking at the root span, not the leaf. Use a structural operator (>>, <<, >, <, ~) to target the right level of the tree.
4. The query that times out. The time window is too long, the attribute has high cardinality, or the querier is starved. Symptom: 30-second timeout on a query that should be sub-second. Inspect tempo_querier_search_results_total and the querier CPU.
5. The query that returns a 403. Multi-tenant deployments enforce X-Scope-OrgID. A query without the header from a Grafana panel whose data source is misconfigured returns unauthorised. Symptom: every Grafana Explore click returns “permission denied”.
6. The query that returns the right answer on Monday and the wrong answer on Tuesday. Schema drift: a new SDK version changes an attribute name (http.url becomes url.full). Symptom: existing dashboards that filter on the old name go empty; engineers assume telemetry is broken when it is the dashboard that is stale.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Does the time window contain any trace at all? Run curl ... | jq '.traces | length' on an empty selector { }. Zero means the time window is empty; the issue is retention or clock skew, not TraceQL.
2. Does the service appear in any trace at all? Run { resource.service.name != nil } and inspect the result set. If the expected service is missing, the SDK is not setting service.name; see lesson 02.
3. Does the attribute exist? Pick one trace ID from the result set above and run curl .../api/traces/{id} | jq on it. The first span shows the full attribute map. Confirm the attribute name and the scope you are filtering on.
4. Does the query syntax parse? The Tempo UI shows a red banner for invalid syntax. A common mistake is forgetting the curly braces or using a reserved word as a literal.
5. Is the querier healthy? curl /querier/ready on the querier pod. A 503 means the ring is unhealthy; check the ingester pods.

Security implications

TraceQL is a structured language: an attacker cannot inject a query through attribute values the way they can inject SQL. The relevant surfaces are:

• Tenant isolation. Every TraceQL search carries the X-Scope-OrgID header. A misconfigured Grafana data source can send the wrong tenant and return traces from another tenant. Symptom: a viewer-role user in one team sees another team’s traces.
• Result size. A TraceQL search without a limit returns as many matches as the engine can hold. A panel that omits the limit is a Grafana OOM waiting to happen. Bound every Grafana panel to a sensible limit.
• Query cost. A user can craft a TraceQL query that scans the entire retention window. querier.max_query_length caps the window; without it, one user can saturate the queriers.
• Audit logging. Tempo logs every TraceQL search at the info level with the query string. Logs with full TraceQL expressions can leak trace ID patterns; pipe them through the same redaction pipeline as application logs.

Performance implications

The cost profile of a TraceQL query is dominated by three factors:

• Time window. A search across 30 days is roughly 60 times more expensive than across 12 hours, because the engine scans every block in the window. Bound the window with max_query_length on the querier and with a Grafana time picker on the panel.
• Scope. A resource. selector reads only the resource column; a span. selector reads the span column. Both are cheaper than reading a nested array (event. or link.). Prefer the broadest scope that still selects the right spans.
• Cardinality of the compared attribute. A filter on a low-cardinality attribute (status, http.method) is fast because the engine can use a bloom filter. A filter on a high-cardinality attribute (http.url, user_id) is slow because every block must be scanned for matches.

The query-frontend cache mitigates all three: a repeated dashboard query hits the cache for up to its TTL and bypasses the querier entirely.

Production guidance

• Start every TraceQL query with the broadest scope that still answers the question. Narrow with && only when the result set is too large to render.
• Pin a max_query_length on every querier. The default is unbounded, which is a denial-of-service waiting to happen.
• Bound the limit on every Grafana panel that issues a TraceQL search. Twenty is a reasonable default; ten for the Explore UI.
• Run the query-frontend in microservices mode. The operational gain over a single querier is large.

Verification

You should now be able to answer:

• What three pieces make up a TraceQL expression?
• What is the difference between a TraceQL search and a trace-by-id lookup?
• Why does a resource. selector typically cost less than a span. selector on the same query?
• What is the role of max_query_length on the querier?
• What is the first thing to check when a TraceQL search returns zero results?

Quiz