Tempo Performance Tuning

A trace query took 45 seconds. The on-call engineer spent an hour tuning the querier pod’s CPU limits and adding another replica. The query still took 45 seconds. The actual cause was that the compactor had not run in 12 days and the bucket held 8 million blocks. The querier’s ListObjectsV2 call was the dominant cost; the CPU was idle waiting for the bucket. The fix was to restore the compactor, drain the backlog, and tune the query-frontend cache for the future.

This lesson is the discipline of Tempo performance tuning: the diagnostic order for slow queries, the most common cause, and the knobs that actually move query latency.

What it is

Tempo performance tuning is the discipline of identifying which component is the bottleneck and applying the right knob. Most slow-query cases are not query-side problems; they are bucket- side problems.

Three layers can be the bottleneck:

• Bucket. The querier lists blocks per query. A bucket with millions of objects spends most of its query time in ListObjectsV2 calls.
• Ingester. A head-block-heavy ingester takes longer to answer trace-by-ID queries against recent data. The querier reads the head block first, then falls back to compacted blocks.
• Querier. A querier pool that is undersized for the query load queues requests. The query latency rises because the queue grows, not because any single query is slow.

The diagnostic order matters. Tuning the querier when the bucket is the bottleneck is the most common mistake.

Why a sysadmin cares

Three operational pains are specific to query performance:

1. Adoption drops. Engineers stop using Tempo when queries are slow. They revert to logs and metrics, which loses the cross-service correlation that traces provide.
2. Investigation time rises. A 45-second query is not a one-off cost; it is the cost of every investigation. Engineers learn to avoid the trace UI; MTTR rises.
3. Storage bill grows. A common reaction to slow queries is to add more querier pods. More queriers do not fix a bucket problem; they just spend more CPU waiting on the bucket.

The right approach is to identify the bottleneck first and apply the matching knob.

How it works

A TraceQL query flows through three layers:

  Grafana
     |
     v
  +---------------------------+
  | query-frontend            |
  |   split, cache, fan-out   |
  +---------------------------+
     |
     v
  +---------------------------+
  | querier                   |
  |   TraceQL execution       |
  |   list blocks (bucket)    |
  |   fetch candidate blocks  |
  |   merge results           |
  +---------------------------+
     |
     | (for recent data)
     v
  +---------------------------+
  | ingester (head blocks)    |
  +---------------------------+
     |
     v
  +---------------------------+
  | object storage            |
  +---------------------------+

The query latency budget for a healthy Tempo cluster, end-to- end, is roughly:

• Query-frontend overhead. 1-5 ms for cache hits; 10-50 ms for cache misses.
• Querier list call. 10-100 ms for a bucket with hundreds of thousands of blocks; 1-10 seconds for a bucket with millions of blocks.
• Querier fetch. 50-500 ms per candidate block, depending on block size and network.
• Querier merge. 1-10 ms.
• Head-block read. 10-100 ms for recent data.

The dominant cost in a healthy cluster is the querier list call. When that grows past 100 ms, the bucket is the bottleneck.

How to configure it

A production querier config pins the query concurrency and the search parameters. The query-frontend config pins the cache:

querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095
  max_concurrent_queries: 200
  search:
    max_concurrent_queriers_per_query: 4

query_frontend:
  max_concurrent_queries: 200
  results_cache:
    cache:
      embedded_cache:
        max_size_items: 1024
        ttl: 1h

compactor:
  compaction:
    block_retention: 48h
    compaction_window: 1h

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1

Three details to call out:

• max_concurrent_queries caps the number of queries a querier pod serves at once. A pod with 4 vCPU and 8 GiB handles roughly 50-100 concurrent queries before the CPU saturates. Beyond that, queries queue.
• max_concurrent_queriers_per_query caps the number of parallel subqueries the querier fans out across. A value of 4 means each query is split into four parallel subqueries. Higher values help when the bucket is the bottleneck; lower values help when the bucket is healthy.
• results_cache.ttl is the query-frontend cache TTL. A value of 1h is the production default. A TTL of 0 disables the cache. A TTL of 24h returns stale results after a config change.

Severity: CONFIGURATION. Reload requires a process restart for most settings.

How to validate it

Severity: READ-ONLY.

1. Confirm query latency percentiles:

curl -s http://tempo-querier:3200/metrics \
  | grep tempo_querier_query_seconds
# tempo_querier_query_seconds_bucket{le="1"} 12345
# tempo_querier_query_seconds_bucket{le="5"} 14200
# tempo_querier_query_seconds_bucket{le="+Inf"} 14302

A le="1" count close to the le="+Inf" count means most queries are fast. A le="5" count close to the le="+Inf" count means most queries take more than 5 seconds.

2. Confirm blocks per query:

curl -s http://tempo-querier:3200/metrics \
  | grep tempo_querier_search_blocklist_latency_seconds
# tempo_querier_search_blocklist_latency_seconds_bucket{le="0.1"} 11000
# tempo_querier_search_blocklist_latency_seconds_bucket{le="+Inf"} 14200

A le="0.1" count close to the le="+Inf" count means most list calls are fast. A flat distribution across buckets means the list call is the bottleneck.

3. Confirm the query-frontend cache hit rate:

curl -s http://tempo-query-frontend:3200/metrics \
  | grep tempo_query_frontend_cache
# tempo_query_frontend_cache_hits_total 8934
# tempo_query_frontend_cache_misses_total 1245

A hit-to-miss ratio of 7:1 is healthy. A ratio of 1:1 means the cache is not effective.

4. Confirm the bucket block count:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix 'blocks/' \
  --max-items 0 \
  --query 'Length'
# 482301

A block count past 1 million is a query-latency warning.

5. Confirm the compactor is keeping up:

curl -s http://tempo-compactor:3200/metrics \
  | grep tempo_compactor_blocks_compacted_total
# tempo_compactor_blocks_compacted_total 4231

A flat counter for hours means the compactor is wedged and the bucket is growing.

How it can fail

Six shapes appear repeatedly:

1. Compactor wedged. The bucket grows monotonically. The querier list call latency rises. Symptom is tempo_querier_search_blocklist_latency_seconds p99 rising alongside tempo_compactor_blocks_compacted_total flat.
2. Querier pool undersized. Query latency rises but ingest is healthy. The querier CPU is near saturation. Symptom is tempo_querier_query_seconds p99 rising and the querier CPU pegged.
3. Query-frontend cache disabled. Every query is a fresh execution. The querier load is higher than necessary. Symptom is the cache miss counter equal to the total request counter.
4. Block-format drift after a binary upgrade. The querier cannot read blocks in the new format and falls back to scanning the older format. Symptom is the querier logging unknown block format and serving slow queries against recent data.
5. Network latency to the bucket. Cross-region bucket access adds 50-200 ms to every block fetch. Symptom is tempo_querier_fetch_duration_seconds p99 rising alongside the bucket’s region.
6. Query too broad. A TraceQL query without a time range filter scans every block in the bucket. Symptom is a single user query consuming the querier’s concurrency budget for minutes.

How to troubleshoot it

The diagnostic order, cheapest first:

1. What is the query p99 latency? tempo_querier_query_seconds.
2. What is the list-call latency? tempo_querier_search_blocklist_latency_seconds.
3. Is the compactor running? /ready and the compactor metric.
4. What is the bucket block count? aws s3api list-objects-v2.
5. Is the querier CPU saturated? kubectl top pod.
6. Is the query-frontend cache effective? Cache hit/miss ratio.

If the list-call latency is the dominant cost, the bucket is the bottleneck. Fix the compactor. If the querier CPU is saturated, add pods. If the cache hit rate is low, tune the cache TTL or the query patterns.

Security implications

The querier has two attack surfaces:

• Query rate limiting. A single user running a broad TraceQL query can consume the querier’s concurrency budget. A max_concurrent_queries cap is the simplest guard; per-user limits are better.
• Trace data exposure. The querier returns trace contents. Authentication is required; unauthenticated access leaks every trace in the tenant.

Performance implications

Query performance has four knobs:

• Compactor. Keeps block count bounded. The dominant fix.
• Querier pool size. Caps concurrency. Add pods for more concurrent queries.
• Query-frontend cache. Reduces querier load for repeated queries. Tune TTL for the query patterns.
• Search parameters. max_concurrent_queriers_per_query caps the per-query fan-out. Higher values help when the bucket is the bottleneck.

The right tuning depends on the bottleneck. Measure first; tune second.

Production guidance

• Alert on tempo_querier_query_seconds p99. A p99 over 5 seconds is a regression.
• Alert on tempo_querier_search_blocklist_latency_seconds p99. A p99 over 200 ms means the bucket is the bottleneck.
• Alert on the bucket object count. A count past 1 million is a query-latency warning.
• Run the compactor. The single most common Tempo performance problem is a stopped compactor.

Verification

You should now be able to answer:

• What is the most common cause of slow Tempo queries?
• What is the diagnostic order for a slow query?
• Which knob is the dominant fix for a bucket-side bottleneck?
• When is adding querier pods the right fix?
• Why does the query-frontend cache matter for query performance?

Quiz