Tempo Architecture Overview

At 03:14 a checkout trace shows a 4.2 s wait on payment-svc. The on-call engineer opens Grafana, searches for the trace ID, and gets the answer in under a second. Behind that single click is a Tempo cluster with at least four distinct services, a write path that batched spans through a ring of ingesters, an object-store bucket holding the flushed block, and a querier that scanned several terabytes of trace data to rebuild the trace from its parts.

This lesson describes that whole system: the components Tempo is made of, the deployment modes the binary supports, and the role of object storage. The five lessons that follow drill into each component.

What it is

Tempo is a distributed tracing backend. It receives spans, batches them into blocks, flushes those blocks to object storage, and serves them back as traces and as TraceQL query results. Like Loki, Tempo is index-less in its default configuration: the trace ID and its block are sufficient to retrieve the trace. The trade-off is unpredictable search; the upside is cheap, durable storage and the ability to keep weeks of trace data without indexing cost.

Why a sysadmin cares

Three operational pains push teams towards Tempo:

1. Index cost. Jaeger and Zipkin rely on a search index that grows with the cardinality of the trace data. At the volumes a mid-sized platform produces, the index is the expensive part of the bill. Tempo removes it.
2. Disk pressure on the ingesters. Because the long-term store is object storage, ingester disk is bounded by the block flush cadence rather than by retention. The trade-off is a write-ahead log on every ingester host.
3. Trace-derived metrics. Service graphs and span-derived RED metrics turn traces into Prometheus metrics. A platform team can derive rates and latencies per service without asking every service team to instrument Prometheus.

The cost is operational complexity. Tempo has more components than Jaeger’s monolithic binary and the failure modes are different.

How it works

A trace enters Tempo through one of three receivers (OTLP, Jaeger, or Zipkin). From there it travels through a fixed pipeline:

  Application / Collector
           |
           v
   +---------------+
   |  Distributors |  validate, rate-limit, fan out
   +-------+-------+
           |
           v  (consistent hash on trace_id)
   +---------------+
   |   Ingesters   |  batch spans into head blocks
   +-------+-------+  flush WAL to local disk
           |
           v  (background flush)
   +---------------+
   |   Object      |  S3, GCS, Azure Blob, MinIO,
   |   Storage     |  or local filesystem backend
   +---------------+
           ^
           |
   +-------+-------+
   |   Queriers    |  TraceQL execution
   +-------+-------+  read blocks from object storage
           |
           v
   +---------------+
   |   Grafana     |  investigate
   +---------------+

The components in that picture, in order:

• Distributor. Receives spans over OTLP / Jaeger / Zipkin, validates them, applies per-tenant rate limits, and forwards them to the correct ingester using a consistent hash on the trace ID.
• Ingester. Stateful. Accumulates spans for a given trace into an in-memory head block, writes the head block to a write-ahead log (WAL) on local disk, and flushes completed blocks to object storage in the background.
• Querier. Stateless. On a TraceQL or GET /api/traces/{id} request, fetches the relevant blocks from object storage, reconstructs the traces, and returns them. TraceQL is GA.
• Query-frontend. Optional. Splits long TraceQL queries, caches sub-results, and parallelises work across queriers. In small clusters this role is left empty; in larger clusters it amortises query cost.
• Compactor. Stateless. Walks the object store, merges small blocks into larger ones, and (in v2) rewrites blocks so that traces with shared IDs are deduplicated.
• Metrics-generator. Reads spans as they pass through the ingester and derives Prometheus metrics: service graphs and span-derived RED metrics.
• Object storage. The long-term store. Tempo treats it as a content-addressed keyspace: blocks are keyed by tenant, block start/end, and an internal version. S3, GCS, Azure Blob, MinIO, and a local filesystem backend are all supported.

Deployment modes

Tempo ships one binary that can play any combination of the six roles. Three deployment modes cover almost every production topology:

Mode             Components              When to use
----------------  ----------------------  --------------------------------
Monolithic       All roles, one binary   Dev, lab, very small prod
Simple scalable  Ingester, querier +     Mid-scale single-cluster prod
                 everything else shared
Microservices    Each role its own       Large or multi-tenant prod
                 process, ring-aware

Monolithic. All six roles run in a single process. The ingester uses local filesystem state and the storage backend is the same box’s disk. This is fine for a lab and for a single-team prod at low volume. It does not survive a process restart cleanly because the WAL lives on the local disk.

Simple scalable. The ingester runs as a stateful set, every other role runs as a stateless deployment, and object storage is external. This is the standard recommendation for a single-cluster production deployment at moderate volume. The binary is the same; the deployment topology differs.

Microservices. Each of the six roles is its own stateful or stateless workload, the distributor and querier register with a ring, and the ingester forms its own ring with replication. This is the right choice for multi-tenant deployments or for clusters that need to scale one role independently of the others. The operational cost is significant: every role needs its own monitoring, capacity plan, and runbook.

The role of object storage

Tempo stores blocks in object storage, keyed by tenant and block window. A block is a directory containing a meta.json file (blocks metadata, version, tenant, time range), a data.parquet file (spans), and one or more index files (bloom filters and per-column index structures). Block windows default to about 15 minutes in monolithic and to the configured flush interval in distributed modes.

The choice of object storage backend is the single largest production decision:

• S3 / GCS / Azure Blob. The default. Cheap, durable, well understood. Lifecycle policies control retention.
• MinIO. Self-hosted S3-compatible storage. Use it when the cloud provider’s bucket is not an option.
• Local filesystem. For labs only. No replication, no durability guarantees across host failure. Never use this for production.

The querier pulls block metadata with a ListObjectsV2 call and downloads only the blocks that intersect the query window. With millions of blocks this scan is the dominant query cost; the compactor’s job is to keep the block count bounded.

How to configure it

A production Tempo configuration pins every role. The minimal end-to-end config for a small cluster using S3 looks like:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: '0.0.0.0:4317'
        http:
          endpoint: '0.0.0.0:4318'
    jaeger:
      protocols:
        grpc:
          endpoint: '0.0.0.0:14250'
        thrift_http:
          endpoint: '0.0.0.0:14268'
    zipkin:
      endpoint: '0.0.0.0:9411'

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3
    heartbeat_period: 5s
    join_after: 10s
    observe_period: 10s

compactor:
  compaction:
    block_retention: 48h     # blocks older than this are deleted
    compaction_window: 1h    # 1-hour compaction windows

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 100
      queue_depth: 10000
  metrics:
    backend: s3
    s3:
      bucket_name: tempo-metrics-prod
      region: eu-west-1

Two production details to call out:

• ingester.replication_factor: 3 requires at least three ingester pods. Two is not enough for the write quorum to succeed.
• compactor.compaction.block_retention: 48h deletes blocks older than two days. Pair this with the bucket’s S3 lifecycle policy for the final delete; the compactor removes the index entry, the lifecycle policy removes the bytes.

How to validate it

After applying the config and starting the binary, the validation order is:

1. Confirm the process is up. The HTTP listen port answers:

curl -s http://tempo.internal:3200/ready
# ready

2. Confirm every role that should be running reports healthy. Tempo exposes a /status endpoint per role path:

curl -s http://tempo.internal:3200/distributor/ready
curl -s http://tempo.internal:3200/ingester/ready
curl -s http://tempo.internal:3200/querier/ready
curl -s http://tempo.internal:3200/compactor/ready
# ready
# ready
# ready
# ready

3. Confirm a trace actually round-trips. Send a single span and retrieve it by ID:

# Send a small OTLP trace via otel-cli
otel-cli span export --endpoint tempo.internal:4317 \
  --service checkout --name POST /charge \
  --attrs payment.method=card

# Search for the trace ID that otel-cli prints
TRACE_ID=...
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | jq .

4. Confirm blocks land in the bucket:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix 'blocks/ingester/' \
  --max-items 5 | jq '.Contents[].Key'
# "blocks/ingester/tenant/01H.../01H...-01H...-meta.json"

5. Confirm TraceQL works against the recent data:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" }' \
  --data-urlencode 'limit=10' | jq '.traces | length'

How it can fail

Six shapes appear repeatedly:

1. Object storage credentials expire or rotate. The ingester accepts writes, the flush to S3 fails, the WAL grows until the disk fills, and the ingester stops accepting new traces. Symptom is tempo_ingester_failed_flushes_total rising alongside disk pressure on the ingester host.
2. Ingester ring is unhealthy. A split-brain between memberlist nodes produces duplicate ingester entries; the distributor forwards spans to instances the ring considers down; spans are accepted but never flushed. Symptom is increasing tempo_ingester_local_checkpoint_manager_last_saved_timestamp age.
3. Wrong backend configured for traces vs metrics. The metrics-generator writes service-graph metrics to a separate bucket. Misconfiguring storage.metrics to point at the trace bucket produces a stream of failed uploads. Symptom is tempo_metrics_generator_registry failing to register registries.
4. Compactor is not running. Block count grows without bound. After hours the querier spends most of its time in ListObjectsV2 calls. Symptom is tempo_compactor_blocks_marked_for_deletion stuck at zero and tempo_querier_blocks_inspected_total rising.
5. Receiver port not exposed. A common post-deployment error is to start the binary without --config.file and miss distributor.receivers. Clients see connection refused. Symptom is tempo_distributor_spans_received_total flat at zero.
6. Auth disabled in production. auth_enabled: false lets any caller who reaches the HTTP endpoint read traces. Symptom is 401 missing on /api/search and on /api/traces/.

How to troubleshoot it

The diagnostic order matters. Each step rules out a layer of the stack:

1. Is the binary running? Check the process, the systemd unit, or the pod status. A crashed process is the obvious case; a process that started but cannot bind to a port is less obvious.
2. Is the process doing what I configured? Hit /ready and /status per role. A ready process that has registered the wrong role is a config bug, not a runtime bug.
3. Can the process reach the bucket? Run aws s3 ls from the Tempo host using the same credentials. An IAM policy that works for the AWS CLI may not work for Tempo because the SDK uses different endpoints.
4. Are traces arriving? Check tempo_distributor_spans_received_total. If the counter is flat, the problem is on the client side, not Tempo.
5. Are blocks flushing? Check tempo_ingester_local_blocks (number of blocks in memory) and tempo_ingester_local_checkpoint_manager_last_saved_timestamp age. A WAL that has not been checkpointed in an hour points at object storage failures.
6. Are queries returning? Check tempo_querier_query_result_size_bytes and the duration histogram. A query that returns empty results is not the same as a query that times out.

Security implications

Three surfaces deserve attention:

• Authentication. Set auth_enabled: true and front the cluster with an authenticating reverse proxy (Grafana Cloud Fleet Management, nginx with JWT, or the Grafana data source proxy). Tempo does not implement authentication of its own; unauthenticated access to /api/traces/{id} leaks every trace.
• Network exposure. The OTLP gRPC and HTTP ports (4317, 4318) and the Jaeger ports (14250, 14268) must not be on a public network. Tempo accepts unauthenticated span payloads. Treat those ports as service-internal.
• Object storage. The S3 bucket should be private. A leaked AWS key with s3:PutObject and s3:GetObject on the Tempo bucket is a trace-data breach. Use scoped credentials and rotate them.

Performance implications

The cost profile differs by role:

• Distributor. CPU-bound on JSON / thrift decoding. Memory pressure appears when the rate limit allows more spans than the ingester pool can drain.
• Ingester. Memory-bound on the head block. A trace that arrives early and stays open for hours can grow a head block to gigabytes. The trace_idle_period closes the block.
• Querier. Network- and disk-bound on block fetches. The compactor keeps block size high enough that fetches are efficient. A querier that fetches thousands of blocks per query is over-scanning.
• Compactor. CPU- and network-bound during the compaction window. Runs as a singleton by default; a single compactor is fine until block counts reach hundreds of thousands.
• Metrics-generator. Memory-bound on the in-memory state per registry. Each registry consumes RAM proportional to the cardinality of its labels.

Production guidance

• Start with simple scalable mode. Move to microservices only when one role is provably the bottleneck.
• Pin the ingester WAL to a dedicated, fast disk. NVMe-backed local SSD is the right answer; network-attached storage is not.
• Set block_retention to the longest time you will ever need to query, and configure the bucket lifecycle to match.
• Run exactly one compactor. More than one risks two compactors racing on the same block.

Verification

You should now be able to answer:

• Name the six Tempo components and the boundary between them.
• What is the difference between monolithic, simple scalable, and microservices modes?
• Why does Tempo use object storage instead of an indexed local store?
• What is the role of the WAL on the ingester?
• Which component do you scale first when query latency rises?

Quiz