Tempo — storing and querying traces

Spans have to land somewhere. Tempo is the trace store that most Docker-hosted stacks end up with, mainly because it indexes almost nothing and therefore costs almost nothing to run — the design choice that also explains its one genuinely surprising limitation.

What Tempo is, and what it deliberately is not

Tempo writes trace blocks to object storage (or a local disk) and keeps only a small index. It does not maintain the rich inverted index that Elasticsearch-backed tracing systems do. In exchange, a year of traces costs roughly what a year of compressed logs costs, and there is no cluster of index nodes to operate.

The consequence: Tempo is excellent at “give me trace 4bf92f3577b34da6” and good at “find traces matching this structured query in the last hour”. It is not a free-text search engine over your span attributes.

A minimal deployment

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    # How long a trace block is kept before compaction deletes it.
    block_retention: 336h   # 14 days

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal

# compose.yaml (excerpt)
services:
  tempo:
    image: grafana/tempo:2.9.0
    command: ['-config.file=/etc/tempo/tempo.yaml']
    user: '10001:10001'
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    networks: [observability]
    restart: unless-stopped

volumes:
  tempo-data:

Note the single-dash -config.file. Tempo, like several Grafana Labs services, uses Go’s standard flag package, so --config.file is also accepted but the documentation and every example use one dash.

Local disk or object storage

backend: local is fine for a single host and for learning. It is also a single point of failure and it does not scale past the disk.

For anything you intend to keep, use S3-compatible object storage — including MinIO running as another container on the same host if you have no cloud provider:

storage:
  trace:
    backend: s3
    s3:
      endpoint: minio:9000
      bucket: tempo
      insecure: true          # plaintext to a container on a private network
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
    wal:
      path: /var/tempo/wal

Tempo does not expand ${VAR} in its configuration unless you ask it to, so that file only works with the flag added:

command: ['-config.file=/etc/tempo/tempo.yaml', '-config.expand-env=true']

Without it the literal string ${TEMPO_S3_ACCESS_KEY} is sent to the object store as a key, and the error you get back is an authentication failure that gives no hint about the cause.

The WAL stays on local disk regardless of backend. It holds spans that have been received but not yet flushed into a block, so it must survive a restart to avoid losing the last few minutes.

Querying with TraceQL

TraceQL selects spans by attribute and returns the traces containing them. The braces are a span matcher:

{ resource.service.name = "api" && span.http.response.status_code = 500 }

{ resource.service.name = "api" && duration > 2s }

{ span.db.system = "postgresql" && duration > 500ms }

Three prefixes do most of the work:

Structural operators are what make it more than a filter. >> means “descendant of”, so this finds slow database spans only when they sit under a request to the checkout service:

{ resource.service.name = "checkout" } >> { span.db.system = "postgresql" && duration > 1s }

That question — “is the database slow for this caller” — is one no metric can answer, and it is the reason to keep traces at all.

You can query from the CLI when Grafana is not to hand:

$ curl -sG http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "api" && duration > 2s }' --data-urlencode 'limit=3' | jq '.traces[] | {traceID, rootServiceName, durationMs}'

{
"traceID": "4bf92f3577b34da6a3ce929d0e0e4736",
"rootServiceName": "api",
"durationMs": 4182
}
{
"traceID": "8a3c1f90b21d4e77b0c5a2119e6f3d84",
"rootServiceName": "api",
"durationMs": 2610
}

And fetch one trace whole, which is what you do when a log line hands you an ID:

$ curl -s http://localhost:3200/api/traces/4bf92f3577b34da6a3ce929d0e0e4736 | jq '.batches | length'

5

Sizing retention

Trace storage is predictable in a way log storage is not, because a span is a small, fixed-ish structure rather than an arbitrary string.

A workable estimate: a span with a normal attribute set costs roughly 300–500 bytes compressed in a Tempo block. So:

spans/sec × 400 bytes × 86400 × retention_days = bytes

At 500 spans/sec and 14 days:

500 × 400 × 86400 × 14 ≈ 242 GB

That number is what makes people reach for sampling, which is the next lesson. Before you do, check the two cheaper levers: are you tracing health-check requests (usually pure waste), and are you attaching large attributes such as full request bodies to spans?

When Tempo is up but search finds nothing

1. Confirm Tempo is ready: curl -sf http://localhost:3200/ready.
2. Confirm it received anything: query {} with limit=1 over the last hour.
3. Confirm the attribute names by reading a real trace rather than assuming the semantic convention.
4. Only then suspect the query.

The boundary with monitoring

Tempo can also generate metrics from spans (its metrics-generator produces service-graph and span-metrics series for Prometheus). That is a real and useful feature, and it belongs to the monitoring part of this course rather than here — it is a metrics pipeline that happens to be fed by traces. Keep the mental separation: this lesson is about storing and retrieving the traces themselves.

Knowledge check

Where next

Storage cost is the pressure that pushes every tracing deployment towards sampling. The next lesson covers what sampling buys, and precisely what you give up.