The Querier

A platform team deploys a new dashboard that runs a TraceQL query over the last 24 hours every 30 seconds. Each query scans every block in the bucket. The querier’s CPU saturates, dashboard panels time out, and the on-call investigation slows because every Grafana click now returns 504. The team adds a query- frontend with a 5-minute cache; the problem disappears.

This lesson describes the Tempo querier: the read path, the role of the query-frontend, and the cost of unbounded TraceQL queries.

What it is

The Tempo querier is a stateless service that answers two kinds of request:

1. Trace-by-id lookup. GET /api/traces/{id} returns the full trace with all its spans. The querier asks every ingester for the trace (because the trace may still be in a head block) and fetches the relevant block from the bucket (because the trace may have been flushed).
2. TraceQL search. GET /api/search and GET /api/search/tag/{name}/values return lists of traces and lists of tag values. The querier scans every block that intersects the query window, applies the TraceQL filter, and returns matching traces.

The querier has no local state. It can be restarted, scaled up, or scaled down without loss of data. The only persistent state it touches is the bucket.

Why a sysadmin cares

Three operational pains are specific to the querier:

1. Bucket reachability. The querier fetches blocks from the bucket on every query. An IAM misconfiguration, a network change, or a bucket region failover manifests at the querier first.
2. TraceQL cost. A TraceQL query that scans 24 hours of blocks is more expensive than one that scans 1 hour. A dashboard that polls a wide query is more expensive than one that polls a narrow query. Without query limits, one Grafana user can saturate the querier pool.
3. Read-after-write consistency. A trace that arrived 30 seconds ago may still be in a head block. The querier asks every ingester for the trace and merges the responses with the bucket result. If the ingester search path is broken, recently arrived traces look missing.

How it works

The querier answers two requests with two paths:

  GET /api/traces/{id}                     GET /api/search
        |                                        |
        v                                        v
  +------------------+                   +-------------------+
  | Query ingesters  |                   | List blocks in    |
  | for trace_id in  |                   | bucket intersecting
  | their head blocks|                   | query window      |
  +--------+---------+                   +---------+---------+
           |                                       |
           |                                       v
           |                             +--------------------+
           |                             | Fetch each block   |
           |                             | from bucket        |
           |                             +---------+----------+
           |                                       |
           |                                       v
           |                             +--------------------+
           |                             | Apply TraceQL      |
           |                             | filter per block   |
           |                             +---------+----------+
           |                                       |
           v                                       v
  +------------------+                   +-------------------+
  | Fetch blocks     |                   | Merge results,    |
  | from bucket      |                   | return trace IDs  |
  | for trace_id     |                   +-------------------+
  +--------+---------+
           |
           v
  +-------------------+
  | Merge head and    |
  | bucket results,   |
  | return full trace |
  +-------------------+

Two production details to call out:

• Trace-by-id merges head and bucket. The querier always asks every ingester for the trace. If an ingester has the trace in a head block, the response wins. If the ingester has already flushed the trace, the bucket result wins. The two responses are merged and deduplicated.
• TraceQL is a scan. A TraceQL query that asks for all traces with status = error over 24 hours scans every block in the window. The cost is proportional to the number of blocks, not the number of matching traces. The compactor bounds block count; the query-frontend bounds query scope.

How to configure it

A production querier config pins the search limits and the worker pool:

querier:
  # Hard limits on the TraceQL search path. Without these, a
  # single unbounded query can saturate the querier pool.
  frontend_worker:
    frontend_address: query-frontend:9095
    parallelism: 10
  max_concurrent_queries: 20
  query_timeout: 30s

  # Block fetch concurrency per query. Higher values use more
  # bucket bandwidth; lower values increase query latency.
  search:
    max_concurrent_requests: 20

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1

query_frontend:
  # The query-frontend is optional. When present, queriers
  # connect to it instead of answering queries directly.
  search:
    max_concurrent_queries: 20
    query_timeout: 30s
  results_cache:
    backend: redis
    redis:
      endpoint: redis.internal:6379
    ttl: 5m

Three production details to call out:

• max_concurrent_queries and query_timeout are the two limits that matter most. A querier pool that accepts 200 concurrent queries with no timeout will saturate on a single wide query.
• The query-frontend cache uses a content-addressed key (tenant, query, time window). A 5-minute TTL is the standard recommendation; shorter TTLs reduce cache hit rate, longer TTLs return stale data.
• The querier’s frontend_address must point at the query- frontend if one is running. A querier that points at the query- frontend is an ingester-side worker; a querier that does not point at one is a direct query endpoint.

How to validate it

Six checks confirm the querier is doing its job:

1. Confirm the querier is ready and registered:

curl -s http://tempo.internal:3200/querier/ready
# ready
curl -s http://tempo.internal:3200/querier/ring | jq .
# {
#   "name": "querier",
#   "members": [{"addr": "tempo-querier-0:3200", "state": "ACTIVE"}]
# }

2. Confirm a trace-by-id lookup returns the trace:

TRACE_ID=...
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | jq '.batches | length'
# 1

3. Confirm a TraceQL search returns matching traces:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" && status = error }' \
  --data-urlencode 'limit=10' | jq '.traces | length'
# 6

4. Confirm the bucket is reachable. The querier logs every block fetch error; the tempo_querier_block_fetch_errors_total counter should remain at zero under healthy operation:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_querier_block_fetch_errors_total
# (no output means zero)

5. Confirm the ingester search path is returning recent traces. A trace that arrived 30 seconds ago should be readable even if it has not flushed:

# Send a trace
otel-cli span export --endpoint tempo.internal:4317 \
  --service test --name probe --attrs marker=recent
TRACE_ID=$(otel-cli span ls --limit 1 --format json | jq -r '.[0].TraceId')

# Wait only briefly so the ingester search path must return it
sleep 1

# Read
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | jq '.batches | length'
# 1

6. Confirm a query that previously scanned many blocks now scans few. The tempo_querier_blocks_scanned_per_query histogram should be dominated by the le="10" bucket under healthy compaction:

curl -s http://tempo.internal:3200/metrics \
  | grep 'tempo_querier_blocks_scanned_per_query_bucket' | head -3
# tempo_querier_blocks_scanned_per_query_bucket{le="1"}  8412
# tempo_querier_blocks_scanned_per_query_bucket{le="10"} 8931

How it can fail

Five shapes appear repeatedly:

1. Bucket unreachable. An IAM policy change, a region failover, or a network route change. The querier cannot fetch blocks. Symptom is tempo_querier_block_fetch_errors_total rising and TraceQL queries returning 500.
2. Unbounded TraceQL query. A user runs a query with no time bound or no result limit. The querier scans every block in the bucket. Symptom is tempo_querier_blocks_scanned_per_query_bucket pushed into the le="+Inf" bucket and CPU saturation.
3. Recent trace not readable. A trace that arrived ten seconds ago is missing from /api/traces/{id}. The ingester search path is broken or the distributor has dropped the trace. Symptom is tempo_ingester_traces_created_total rising while the same trace returns empty from the querier.
4. Query-frontend cache stale. The query-frontend returns cached results that do not include the last few minutes of spans. Symptom is user complaints about “the dashboard is showing old data” despite recent activity.
5. Worker pool saturated. max_concurrent_queries exceeded. New queries wait or time out. Symptom is tempo_querier_concurrent_queries at the limit and Grafana returning 504.

How to troubleshoot it

The diagnostic order:

1. Is the querier ready? Check /querier/ready. A not-ready querier should not be in the load balancer.
2. Can the querier reach the bucket? Run aws s3 ls from the querier host using the same credentials. An IAM policy that works for the AWS CLI may not work for Tempo because the SDK uses different endpoints.
3. Are queries returning? Run a known-good query (small time window, known tag) and check the response. A query that returns empty results is not the same as a query that times out.
4. Is the search path reaching ingesters? Check tempo_querier_ingester_searches_total. A flat counter means the ingester search path is broken; recently arrived traces will be missing.
5. Is a single query dominating the pool? Check tempo_querier_blocks_scanned_per_query_bucket. A query in the le="+Inf" bucket is the source of the saturation.
6. Is the query-frontend caching correctly? Check tempo_query_frontend_results_cache_hits_total and tempo_query_frontend_results_cache_misses_total. A miss ratio above 80% means the cache is too small or the TTL is too short.

Security implications

The querier reads from the bucket and serves traces to Grafana:

• Authentication. Tempo does not implement authentication. Front the querier (or the query-frontend) with a reverse proxy that enforces JWT or mTLS.
• Tenant isolation. Multi-tenant deployments must enforce that one tenant’s query cannot read another tenant’s traces. Tempo uses the X-Scope-OrgID header to scope every request; the querier never trusts the trace ID alone.
• Bucket credentials. The querier reads from the bucket but does not write. A read-only scoped credential is sufficient. A leaked read/write credential is a write attack surface as well as a read attack surface.

Performance implications

The querier is network- and disk-bound on block fetches:

• Network. Each query fetches every intersecting block. A TraceQL query over 24 hours fetches roughly the entire bucket’s metadata. The querier’s bandwidth scales with query count and bucket size.
• Disk. The querier caches block metadata in memory. A bucket with 10 million blocks consumes gigabytes of memory for the metadata cache.
• CPU. The TraceQL filter is cheap per block but expensive in aggregate. A scan of 10,000 blocks consumes more CPU than a scan of 1,000 blocks by an order of magnitude.
• Worker pool. max_concurrent_queries bounds the CPU and network cost. A pool of 20 with 30 s timeouts can answer roughly 40 queries per minute per querier pod.

Production guidance

• Run at least two querier pods. One is a single point of failure.
• Add the query-frontend when query load rises. The cache collapses the cost of dashboards that poll wide queries.
• Set max_concurrent_queries based on querier pod size. 20 is a starting point; raise it on bigger pods, lower it on smaller ones.
• Set query_timeout to the longest legitimate TraceQL query, not to a guess. 30 s is a starting point; raise it for known wide queries, lower it for known narrow ones.

Verification

You should now be able to answer:

• What is the querier’s role in the Tempo read path?
• What is the difference between trace-by-id and TraceQL search?
• How does the query-frontend reduce query load?
• Why must max_concurrent_queries and query_timeout be set explicitly?
• What is the operational effect of an unbounded TraceQL query?

Quiz