Tempo Microservices Mode

At 02:14 a payment-svc trace query returned the right trace in under a second. At 02:47 the same query returned the same trace in twenty-two seconds. The metric tempo_querier_query_seconds jumped with it. The on-call engineer, trained on simple-scalable mode, restarted the distributor pod and watched the metric get worse. The distributor was not the bottleneck. The querier pool was at saturation. The fix was three additional querier pods and a query-frontend in front of them, both deployed with target: querier and target: query-frontend.

This lesson is the entry point for a microservices Tempo deployment: the six roles, the wiring between them, and the order in which to scale them.

What it is

Microservices mode runs each Tempo role as its own process and its own workload. The same tempo binary plays the role selected by the top-level target flag and the YAML keys present. Six roles cover the whole pipeline:

   Role               Stateful?   Scales on                 Default target
   ------------------ -----------  ------------------------  -------------------
   distributor        no          spans per second          distributor
   ingester           yes         spans per second + RF     ingester
   querier            no          concurrent queries        querier
   query-frontend     no          long TraceQL queries      query-frontend
   compactor          no          bucket block count        compactor
   metrics-generator  no          derived RED cardinality   metrics-generator

The first three roles are mandatory. The query-frontend is optional but recommended once the querier pool exceeds two pods. The compactor is mandatory in any deployment that retains data beyond the flush window. The metrics-generator is optional and runs as a sidecar to the querier.

Why a sysadmin cares

Three operational pains push teams from simple-scalable to microservices:

1. One role is the bottleneck and the rest are idle. A platform with heavy ingest but light query load wastes CPU when distributor and querier share a process. Splitting them lets each scale on its own metric.
2. Failure isolation. A crash of the ingester in simple-scalable mode also restarts the querier running in the same process. In microservices only the ingester pod restarts; queriers keep serving from compacted blocks.
3. Multi-tenant isolation. Multi-tenant deployments need per-tenant rate limits at the distributor and per-tenant compaction at the compactor. Both are easier when each role is its own workload with its own config map.

The cost is operational surface. A microservices cluster needs six or more StatefulSets / Deployments, a per-role SLO, and an operator who can read ring state.

How it works

The pipeline stays the same as in simple-scalable mode. The difference is that each box in the picture is its own pod, with its own address, and the wiring between them is explicit.

  Application / Collector
            |
            v
  +-------------------+      +-------------------+
  | distributor x N   | ---> | ingester x N      | ---> S3 / GCS / Azure
  | OTLP / Jaeger /   |      | head blocks + WAL |
  | Zipkin receivers  |      | ring, RF=3        |
  +-------------------+      +-------------------+
                                     ^
                                     | (tail block reads)
            +-------------------+   |
            | query-frontend    |---+ (cache, split, fan-out)
            +-------------------+
                     |
                     v
            +-------------------+
            | querier x N       |
            | TraceQL execution |
            +-------------------+
                     ^
                     |
            +-------------------+
            | compactor (one)   |----> walks S3, merges blocks
            +-------------------+
            +-------------------+
            | metrics-generator |----> service-graph metrics
            +-------------------+

The roles, in order of how often they need attention:

• Distributor. Stateless. Validates spans, applies per-tenant rate limits, hashes the trace ID, and forwards the span to the three ingesters that own that hash range. The distributor is CPU-bound on JSON / thrift decoding.
• Ingester. Stateful. Accumulates spans into head blocks, fsyncs the head block to a WAL on local disk, and flushes completed blocks to object storage. Runs as a StatefulSet with a memberlist or etcd ring KV.
• Querier. Stateless. On a TraceQL or trace-by-ID request, fetches blocks from object storage, reconstructs traces, returns results. CPU- and network-bound on block fetches.
• Query-frontend. Stateless. Splits long TraceQL queries into subqueries, caches subresults in a FIFO cache, and parallelises work across queriers. Required when the querier pool exceeds two pods.
• Compactor. Mostly stateless. Walks the bucket, merges small blocks into larger ones, deduplicates blocks that share trace IDs. Runs as a singleton by default; sharded compaction exists in recent versions but requires explicit configuration.
• Metrics-generator. Stateless. Reads spans from the ingester pipeline and derives Prometheus metrics: service graphs and span-derived RED metrics. Runs as a querier sidecar.

How to configure it

A minimal microservices config pins every role and the wiring between them. The values below match the official Helm chart defaults but are written out for inspection:

server:
  http_listen_port: 3200

# Shared by all roles.
distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: '0.0.0.0:4317' }
        http: { endpoint: '0.0.0.0:4318' }
    jaeger:
      protocols:
        grpc:    { endpoint: '0.0.0.0:14250' }
        thrift_http: { endpoint: '0.0.0.0:14268' }
  ring:
    kvstore:
      store: memberlist

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3
    heartbeat_period: 5s
    join_after: 10s
    observe_period: 10s

querier:
  frontend_worker:
    frontend_address: tempo-query-frontend:9095

query_frontend:
  max_concurrent_queries: 200
  results_cache:
    cache:
      embedded_cache:
        max_size_items: 1024
        ttl: 1h

compactor:
  compaction:
    block_retention: 48h
    compaction_window: 1h
  ring:
    kvstore:
      store: memberlist

metrics_generator:
  registry:
    external_labels:
      source: tempo

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
    wal:
      path: /var/tempo/wal
  metrics:
    backend: s3
    s3:
      bucket_name: tempo-metrics-prod
      region: eu-west-1

Three details to call out:

• distributor.receivers lists every receiver the process exposes. In microservices mode each distributor pod exposes all receivers; a service load-balances the OTLP and Jaeger ports.
• querier.frontend_worker.frontend_address must point at the query-frontend DNS, not localhost. A common post-migration bug is leaving the simple-scalable default.
• compactor.ring is mandatory when sharded compaction is enabled. Without the ring the compactor still works as a singleton.

How to validate it

Severity: READ-ONLY.

1. Confirm each role is ready in its own pod:

for role in distributor ingester querier query-frontend compactor metrics-generator; do
  echo -n "$role: "
  curl -s "http://tempo-${role}.observability.svc:3200/ready"
  echo
done
# distributor: ready
# ingester: ready
# querier: ready
# query-frontend: ready
# compactor: ready
# metrics-generator: ready

2. Confirm the ingester ring has the right membership:

curl -s http://tempo-ingester:3200/ingester/ring | jq '.members | length'
# 3
curl -s http://tempo-ingester:3200/ingester/ring \
  | jq '.members[] | {addr, state}'
# {"addr": "tempo-ingester-0:3200", "state": "ACTIVE"}
# {"addr": "tempo-ingester-1:3200", "state": "ACTIVE"}
# {"addr": "tempo-ingester-2:3200", "state": "ACTIVE"}

3. Confirm spans arrive at the distributor and reach the ingester:

curl -sG http://tempo-distributor:3200/metrics \
  | grep tempo_distributor_spans_received_total
# tempo_distributor_spans_received_total{tenant="single-tenant"} 12842

4. Confirm the querier answers TraceQL:

curl -sG http://tempo-querier:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" }' \
  --data-urlencode 'limit=5' | jq '.traces | length'
# 5

5. Confirm the compactor is making progress on the bucket:

curl -s http://tempo-compactor:3200/metrics \
  | grep tempo_compactor_blocks_compacted_total
# tempo_compactor_blocks_compacted_total 4231

How it can fail

Six shapes appear repeatedly in microservices mode:

1. Role address drift after a Helm upgrade. A chart rename changes the service name (tempo-query-frontend to tempo-queryfrontend). Queriers keep the old DNS in their config and lose contact. Symptom is the querier returning no frontend errors and trace-by-ID lookups succeeding only when the request lands on a pod that can reach the frontend.
2. Ingester ring unstable across replicas. A misconfigured join_after or heartbeat_timeout causes ingesters to flap between JOINING and ACTIVE. Symptom is tempo_ingester_lifecycler_ring_inconsistencies_total rising and the distributor returning 503.
3. Compactor missing. The binary tolerates its absence, the bucket does not. Symptom is tempo_compactor_blocks_compacted_total flat at zero while the bucket grows. The bill arrives first, the query latency second.
4. Querier pool too small. Query latency rises but ingest is healthy. Symptom is tempo_querier_query_seconds p99 growing and the querier CPU near saturation. The fix is pods, not config.
5. Query-frontend cache misconfigured. A TTL of 0s makes the cache useless; a TTL of 24h returns stale results after a config change. Symptom is the same query returning different results across two consecutive Grafana clicks.
6. Metrics-generator attached to the querier but the querier has no ingester connection. Service-graph metrics are missing for new tenants. Symptom is the tempo_metrics_generator_registry failing to register and service_graph_request_total flat at zero.

How to troubleshoot it

The diagnostic order matters. Each step rules out one role:

1. Is every role ready? /ready per role. A 404 points to a wrong target flag in the unit file or to a Helm value that disables the role.
2. Is the ring healthy? Hit /ingester/ring and /distributor/ring. An empty members list means the KV store (memberlist or etcd) is not reachable between pods.
3. Are spans arriving? tempo_distributor_spans_received_total and tempo_ingester_spans_received_total. A flat distributor counter is a client-side problem; a flat ingester counter is a wiring problem.
4. Are blocks flushing? tempo_ingester_local_blocks and the checkpoint timestamp. Stale checkpoints point at object storage failures.
5. Is the bucket compacting? tempo_compactor_blocks_compacted_total. A flat counter for hours means the compactor is wedged.
6. Is the query path responsive? tempo_querier_query_seconds and tempo_query_frontend_queries_total. p99 latency over five seconds almost always points at the bucket, not the querier.

Security implications

Microservices mode exposes more ports because each role has its own service. Three surfaces:

• Per-role addresses become stable DNS names. Bind every role’s HTTP and gRPC listen address to a private interface. Treat the OTLP, Jaeger, and Zipkin ports as service-internal.
• mTLS between roles. The ring KV and the querier-to-query-frontend link are cluster-internal but should not be plaintext in a shared cluster. Use cert-manager or your platform’s mTLS mesh.
• Bucket credentials. The ingester and compactor both write to the bucket. A leaked AWS key with s3:PutObject is a trace-data breach. Use scoped credentials per role and rotate them.

Performance implications

The cost profile differs by role:

• Distributor. CPU-bound on decoding. Memory grows with the rate-limit queue.
• Ingester. Memory-bound on the head block; disk-bound on the WAL.
• Querier. CPU- and network-bound on block fetches.
• Query-frontend. Memory-bound on the in-memory cache; CPU on query splitting.
• Compactor. CPU- and network-bound during the compaction window.
• Metrics-generator. Memory-bound on the in-memory registry per service.

The right scale order, when one role is the bottleneck:

1. Querier. Read load grows with the number of Grafana users and dashboards. Add pods first when p99 query latency rises.
2. Distributor. Write load grows with the number of instrumented services. Add pods when CPU is saturated and distributor_spans_received_total is still rising.
3. Ingester. Only when the head-block memory ceiling is reached, because each new pod rebalances the ring and may cause a brief 503 window.
4. Compactor. Almost never. Singleton is the default and is sufficient for millions of blocks.
5. Metrics-generator. When the derived-metrics cardinality is the bottleneck, not when the underlying trace volume is.

Production guidance

• Start on simple-scalable mode. Move to microservices when one role is provably the bottleneck.
• Pin the ingester WAL to a dedicated, fast disk. NVMe-backed local SSD is the right answer.
• Run exactly one compactor by default; only enable sharded compaction when the bucket exceeds what a single compactor can process per cycle.
• Front the querier with the query-frontend once the querier pool exceeds two pods.

Verification

You should now be able to answer:

• Name the six Tempo roles and which are stateless, stateful, or optional.
• Which role do you scale first when query latency rises?
• What does the query-frontend add when the querier pool exceeds two pods?
• Why does the compactor run as a singleton by default?
• What is the right ring KV choice for a single Kubernetes cluster, and when does that change?

Quiz