Tempo Querying

An on-call engineer opens a Grafana dashboard for a checkout slowdown incident. The user-impact metric says p99 latency for POST /checkout has tripled in the last 30 minutes. The engineer needs to know: which service is the slow one, on which trace, and at which span. They click “Explore” in Grafana, type { resource.service.name = "checkout-api" && status = error } into the Tempo query bar, and a list of trace IDs appears. They pick the most recent. The trace shows the parent span POST /checkout ran for 4.2 s, of which 4.1 s was a child span called payment-charge against the payments service. The investigation took 90 seconds because Tempo accepted the TraceQL query and returned the answer.

The query path that produced that answer is the subject of this lesson.

What it is

Tempo exposes two distinct query shapes:

• Trace by ID. GET /api/traces/{trace-id}. Returns the full trace. Used by Grafana when the user clicks a trace ID from logs, exemplars, or another trace.
• TraceQL search. GET /api/search?query=... (or the equivalent streaming endpoint). Returns a list of trace IDs matching a TraceQL expression. Used when the user does not know the trace ID and wants to find it.

TraceQL is a SQL-like expression language with two scopes: spans and traces. A span-level filter selects individual spans; a trace-level filter operates on aggregate properties of the whole trace.

Why a sysadmin cares

The two query shapes have different operational profiles:

• Trace-by-ID is a single block read. Latency is bound by the size of one block (a few hundred ms). It scales linearly with concurrent users.
• TraceQL search is a block scan across the time window. Latency is bound by the number of blocks in the window and the size of the time range. A search across 30 days can be several seconds; a search across 30 minutes is sub-second.

The query-frontend exists to make the second shape cheap. The metrics-generator exists to avoid the second shape by pre-computing RED metrics from spans.

How it works

   Grafana
      |
      v
   Tempo query-frontend  <-- splits subqueries, caches results
      |
      v
   Tempo querier  <-- reads blocks, applies TraceQL filter
      |
      v
   Object storage (S3/GCS/Azure)

TraceQL

A TraceQL expression is a filter over the trace tree. The default scope is spans. A query of { resource.service.name = "checkout-api" } selects every span belonging to that service. To operate at the trace level, wrap the filter in a {} aggregate (a trace aggregate):

{ resource.service.name = "checkout-api" && status = error }
  | count() > 0

The | count() > 0 aggregate returns only the trace IDs that have at least one matching span. Without the aggregate, the query returns individual spans.

Two intrinsic fields are available on every span:

• status — ok, error, or unset.
• kind — server, client, producer, consumer, internal.

Plus all attributes set by the SDK (span attributes, resource attributes, events). Examples:

{ resource.service.name = "checkout-api" }                         # service
{ name = "POST /checkout" && http.status_code >= 500 }              # by name
{ duration > 2s }                                                # slow
{ span.http.url =~ ".*/v2/.*" }                                     # URL match
{ resource.k8s.pod.name = "checkout-api-7d4b" }                     # by pod

Aggregates extend the filter:

{ resource.service.name = "checkout-api" } | count() > 0
{ resource.service.name = "checkout-api" } | avg(duration) by (resource.k8s.pod.name)
{ span.http.status_code = 500 } | count() by (resource.service.name) > 5

Search vs lookup

Search reads the bloom filter of every block in the time window. Lookup reads the single block that contains the trace ID.

Query-frontend

The query-frontend is an optional but production-critical component. It receives the Grafana query, splits it into subqueries by tenant and time range, dispatches the subqueries to querier pods, and merges the results. It also caches TraceQL results for short windows.

Without the query-frontend, every query goes to one querier pod. With it, large queries are split and parallelised, and duplicate queries within a cache TTL hit the cache.

Metrics-generator

The metrics-generator derives RED metrics from spans and exports them to a Prometheus-compatible endpoint. This turns “how many traces had an error in the last 5 minutes” from a TraceQL query (slow, every time) into a Prometheus query (rate(tempo_metrics_generator_spans_total{status="error"}[5m]), fast, cached).

The generator runs as a separate process; it tails the ingester ring for new blocks, parses the spans, and emits metrics. It does not replace the source-of-truth block in storage; it produces a derivative signal.

Under the hood

How to configure it

Query-frontend

# /etc/tempo/tempo.yaml
query_frontend:
  address: tempo-query-frontend:9095

  # Split subqueries across multiple queriers.
  split_queries_by_interval: 15m
  max_parallelism: 4

  # Cache search results for 60 seconds. Useful for repeated
  # dashboard queries.
  search_results_cache:
    cache:
      enable_fifo_cache: true
      fifocache:
        size: 1024
        validity: 60s

Severity: CONFIGURATION. The query-frontend must be deployed alongside the querier; it is its own process in microservices mode (target: query-frontend).

Querier

querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095
    parallelism: 4

  # Bound the time range for a single search. Without this, a
  # Grafana user can scan the whole retention period.
  max_query_length: 168h       # 7 days

Severity: CONFIGURATION. Restart the querier to apply.

Metrics-generator

metrics_generator:
  registry:
    collection_interval: 30s
    timeout: 10s
  storage:
    path: /var/tempo/generator/wal
  traces_storage:
    path: /var/tempo/generator/traces
  processor:
    service_graphs:
      dimensions: [http.method, http.status_code]
    span_metrics:
      dimensions: [http.method, http.status_code]

The service_graphs processor produces the data behind the Grafana “Service Graphs” view; span_metrics produces the RED metrics (traces_spanmetrics_*).

Severity: CONFIGURATION. The generator reads from the ingester ring; it must be deployed with target: metrics-generator in microservices mode.

How to validate it

Severity: READ-ONLY.

# 1. TraceQL search — list error traces for a service
curl -sG "http://tempo:3200/api/search" \
  --data-urlencode 'query={ resource.service.name = "checkout-api" && status = error }' \
  --data-urlencode 'limit=5' | jq .

# 2. Trace by ID lookup
curl -s http://tempo:3200/api/traces/4bf92f3577b34da6a3ce929d0e0e4736 | jq .

# 3. Confirm the metrics-generator is exporting
curl -s http://tempo-metrics-generator:3200/metrics | \
  grep tempo_metrics_generator_spans_total

Real output:

$ curl -sG "http://tempo:3200/api/search" \
    --data-urlencode 'query={ resource.service.name = "checkout-api" && status = error }' \
    --data-urlencode 'limit=2' | jq '.traces[].traceID'
"4bf92f3577b34da6a3ce929d0e0e4736"
"7a1f8b9c0d4e3f2a5b6c7d8e9f0a1b2c"

$ curl -s http://tempo-metrics-generator:3200/metrics | \
    grep -E 'tempo_metrics_generator_spans_total\{' | head -3
tempo_metrics_generator_spans_total{service_name="checkout-api",span_name="POST /checkout",span_kind="server",status_code="server",status="ok"} 4321
tempo_metrics_generator_spans_total{service_name="checkout-api",span_name="POST /checkout",span_kind="server",status_code="server",status="error"} 87

How it can fail

1. TraceQL search returns no results despite traces being ingested. The query is wrong (attribute name typo) or the resource scope is mismatched. Symptom: empty result set; tempo_querier_search_results_total non-zero but tempo_querier_search_result_traces_total zero.

2. Search is slow over a 30-day window. No max_query_length set, or set too high. The querier scans every block in the window. Symptom: 30-second query times for an interactive Grafana panel; user complains.

3. Query-frontend splits but the cache misses. The cache size is too small for the dashboard’s query rate. Symptom: query_frontend_results_cache_hits_total flat; querier CPU high.

4. Metrics-generator drift. The generator’s WAL is on a slow disk and falls behind the ingester. Symptom: tempo_metrics_generator_processed_spans_total rate lower than ingest rate; Grafana dashboards show stale data.

5. http.method dimension not set in the generator. The generator emits RED metrics without the dimension; the dashboard cannot break down by method. Symptom: Grafana panel shows a single line; team cannot isolate which method is slow.

6. Querier cannot reach the ingester ring. The querier logs ring error: at least one ingester must be healthy. Symptom: lookup returns “trace not found” even for traces within retention; search returns empty.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Is the query-frontend alive? curl /ready on the query-frontend pod. If 404, the role is not deployed.
2. Is the querier alive? curl /ready on the querier pod.
3. Is the ring healthy? The querier log mentions the healthy ingester set on every search. Empty == ring is unhealthy; check the ingester pods.
4. Is the metrics-generator processing? Compare tempo_metrics_generator_processed_spans_total to the ingester’s tempo_ingester_spans_received_total. A divergence means the generator has fallen behind.

Security implications

• TraceQL injection. TraceQL is structured; an attacker cannot break out of the query syntax with attribute values. But attribute values can be large (a url.full attribute might contain untrusted text). Bound the result size with limit on every Grafana panel.
• Auth. TraceQL queries honour the X-Scope-OrgID header in the same way as ingest. A user with a viewer role in one tenant cannot query another tenant’s traces.
• Metrics-generator egress. The generator exports to a Prometheus-compatible endpoint. In multi-tenant mode, ensure the generator’s scrape target is on a private network.

Performance implications

• TraceQL cost. A search across a long time window is expensive. max_query_length caps the cost at the querier level; split_queries_by_interval on the query-frontend parallelises the cost.
• Cache. The query-frontend cache helps repeated queries (dashboard panels that re-render every 30 seconds). Tune the cache size to the query rate.
• Metrics-generator CPU. Parsing spans and emitting metrics is roughly 5-10% of the cost of running the same data through a Java tracing agent. The generator can keep up with most production ingests on a single core.

Production guidance

• Deploy the query-frontend in microservices mode. The operational gain is large.
• Set max_query_length to the largest window a Grafana panel is allowed to query. 7 days is a reasonable default.
• Run the metrics-generator as its own pod with its own WAL volume. Do not run it as a sidecar of the querier.
• Bound limit on every Grafana panel. An unbounded limit is a Grafana OOM waiting to happen.

Verification

You should now be able to answer:

• What is the difference between a TraceQL search and a trace-by-ID lookup?
• Which two endpoints expose these operations in Tempo?
• What is the role of the query-frontend?
• What does the metrics-generator produce, and where does it emit it?
• How would you bound the time window for a TraceQL search?

Quiz