Tempo Loss

A trace query for a known payment-service trace ID returns Not Found. The trace was definitely emitted — the application logs include the OTel export call, and the collector logs say traces exported: ok. The Tempo ingester fleet is healthy, the querier is up, and the search returns nothing. You check the block backend and find the bucket is gone.

This lesson is the recovery procedure for that incident. Tempo is simpler than Loki — there is no index to compact, no per-tenant rate limiter to misconfigure — but its durability story is the same: the object store is canonical.

What it is

Tempo loss is any condition in which a trace search by trace ID cannot return the truth. Three shapes:

• Ingester loss. The in-memory span buffer is gone. Traces currently buffered in the ingester are lost. Traces already flushed to the block backend are intact.
• Block-backend loss. The bucket holding the parquet blocks is gone. Every trace older than the last successful flush is gone.
• Trace-ID generator loss. A hash-ring generator used by tracing-aware load balancers has lost its state. New trace IDs collide with old ones and the querier returns the wrong trace.

The first job is to identify which shape you have. The recovery procedure depends on it.

Why a sysadmin cares

Tempo is the second stop in an incident. The first stop is metrics; the second is logs; the third is traces. A trace search that returns the wrong trace is worse than one that returns no trace, because the on-call engineer follows the wrong causal chain. A platform whose traces lie is a platform whose incident duration is unbounded.

Recovery is also the place where multi-zone Tempo is confused with cross-region replication. They are not the same. Multi-zone Tempo adds ingest and query redundancy; the bucket is still single-region. A regional outage takes the bucket with it.

How it works

Tempo is a write path and a read path:

   otel-collector / alloy
        |
        v
   distributor (auth, rate limit)
        |
        v
   ingester (per-tenant span buffer)
        |
        v
   block flush ----> object store (parquet + trace-id index)
                              ^
                              |
   querier / query-frontend --- TraceQL ---> object store

The ingester buffers spans per tenant and flushes them as columnar parquet blocks on a configurable boundary. The querier reads blocks from the same backend. There is no separate index service in the default single-tenant deployment; the trace-ID index lives inside the blocks themselves.

Under the hood

The block backend is the only long-term home for traces. There is no per-tenant state on the querier that survives a flush; the querier is stateless.

How to configure it

A Tempo that is built to be rebuilt has the block backend replicated cross-region, the ingester WAL or equivalent buffer durable across restarts, and the querier pointed at the same backend across zones.

# /etc/tempo/tempo.yaml
server:
  http_listen_port: 3100

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  max_block_bytes: 500_000_000
  complete_block_timeout: 15m

compactor:
  compaction:
    block_retention: 744h
    compacted_block_retention: 720h

storage:
  trace:
    backend: s3
    s3:
      bucket: eu-west-1-tempo-blocks
      endpoint: s3.eu-west-1.amazonaws.com
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
    wal:
      path: /var/tempo/wal
  metrics:
    backend: prometheus

The wal.path is what carries the unflushed window across an ingester restart. Without it, every restart loses up to max_block_duration of traces.

How to validate it

Three checks, in order, before declaring recovery complete:

# 1. The ingester has finished replay and is ready.
curl -sf http://tempo:3100/ready | head -1
# expected: ready

# 2. The querier can reach the block backend.
curl -s http://tempo:3200/api/echo
# expected: empty 200

# 3. A known trace ID returns the trace.
# Pick a trace ID from a recent log line or from the OTel collector export log.
TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo:3200/api/traces/${TRACE_ID}" \
  | jq '.batches | length'

2

If the search returns zero batches, the trace is not in the block backend. The recovery is not done.

How it can fail

• Bucket deleted, and the cross-region replication destination is empty because replication was never enabled. Recovery is impossible. The RPO is whatever the last backup held.
• Ingester WAL on the same volume as the querier cache. A volume loss takes both the unflushed window and the in-memory query cache, which is acceptable, plus any cache the querier was warming, which is not.
• Compactor restarted during a block compaction. The half-built block prefix is left behind. Manual cleanup of the prefix is required before the next compaction cycle.
• Trace-ID generator reset after a database loss. New IDs collide with old IDs for a window. Trace searches return traces with the right ID and the wrong context.
• Per-tenant block quota set too low during recovery. The ingester rejects spans and silently drops the tail of the trace.
• Recovery performed against a destination bucket in a region different from the collector’s region. Cross-region latency inflates the ingest path and overruns the WAL.

How to troubleshoot it

The diagnostic order:

1. Is the ingester running? (kubectl get pods -l app=tempo, component=ingester)
2. Is the WAL replay finished? (logs say wal replayed)
3. Is the querier reachable? (/ready on port 3200)
4. Is the block backend reachable? (/api/echo from the querier)
5. Does a known trace ID return a trace? (/api/traces/:id)
6. Is the compactor caught up? (/compactor/status)

Security implications

The block backend holds every trace the platform retained. Trace data often contains request payloads, header values, and database query strings. A leaked or lost bucket is a compliance event, not just a durability event. Encryption at rest is mandatory.

The trace-ID generator, when in use, holds a per-tenant counter that can correlate IDs to originating services across regions. Treat it as a credential: scope to one Tempo, rotate on schedule.

Performance implications

Block flush scales with the unflushed window. A 30-minute block interval means the WAL can carry up to 30 minutes of spans per tenant. The cold-start replay scales with the WAL size and the number of distinct trace IDs in it.

Compactor cycles scale with the block count, not the trace volume. A platform with many small blocks has a longer compaction window than a platform with fewer large blocks.

Production guidance

• Enable cross-region replication on the block backend. Multi-zone Tempo is not a substitute.
• Size the WAL for the worst-case flusher outage. The WAL is the only place the unflushed window exists.
• Test the recovery by deleting one ingester pod and verifying that a known trace ID is still searchable. This proves the backend is canonical.

Verification

You should now be able to answer:

• What three loss shapes does a Tempo recovery distinguish, and what is the right first action for each?
• Why does the WAL need its own dedicated volume, and what does it carry across restarts?
• What is the right canary query to confirm a recovered Tempo is serving historical blocks?
• Why does multi-zone Tempo not protect against a regional bucket failure?

Quiz