The Ingester

The trace-by-id lookup is the headline feature of Tempo. The on-call engineer pastes a trace ID into Grafana and gets the trace back, regardless of which ingester happened to flush it. That property depends on the ingester doing four things at once: batching spans into a head block, persisting the head block to a WAL on local disk, flushing the block to object storage in the background, and answering trace-by-id queries out of the head block while it is still open. Get any one of these wrong and the property breaks.

This lesson describes the Tempo ingester: the trace lifecycle, the head block, the WAL, and the flush cadence.

What it is

The Tempo ingester is a stateful service. It is the only component in Tempo that holds spans in memory and writes to a local write-ahead log. Its responsibilities are four:

1. Accumulate spans into head blocks, keyed by trace ID.
2. Persist each head block to a WAL on local disk so unflushed spans survive a process restart.
3. Flush completed blocks to object storage in the background.
4. Serve trace-by-id queries out of the head block until the block is flushed.

The ingester runs as a StatefulSet because each instance owns a fixed range of trace IDs. Two ingesters both serving the same trace ID would either duplicate writes or fight on the ring.

Why a sysadmin cares

Three operational pains are specific to the ingester:

1. Disk pressure from the WAL. A healthy ingester flushes blocks faster than the WAL grows. A broken object storage credential or a slow bucket causes the WAL to grow until the disk fills, at which point the ingester stops accepting new spans. The investigation requires recovering the WAL or accepting the loss.
2. Memory pressure from long-lived traces. A trace that arrives in the first millisecond of a window and stays open for thirty minutes lives in a head block for the entire window. A burst of long-lived traces can exhaust ingester memory even when ingest rate is normal.
3. Ring instability during rolling restarts. A StatefulSet rolling update re-elects ingester ownership of trace ID ranges. If the lifecycler is misconfigured, the ring churns, spans go to instances that think they own a different range, and the distributor receives 503 from the ingester pool.

How it works

The trace lifecycle from first span to flushed block:

  Span arrives at distributor
            |
            v
  Distributor hashes trace_id → ingesters in ring
            |
            v
  +--------------------------------------------+
  |                  Ingester                  |
  |                                            |
  |   Span  →  trace_id lookup            |
  |                                            |
  |   new trace_id   →  create head block |
  |   known trace_id →  append to block   |
  |                                            |
  |   every flush_period:                      |
  |     head block  →  WAL append         |
  |                                            |
  |   on trace_idle_period exceeded:           |
  |     head block  →  completed block   |
  |                                            |
  |   on max_block_duration exceeded:          |
  |     force flush oldest block               |
  |                                            |
  |   background flush loop:                   |
  |     completed block  →  object store |
  +--------------------------------------------+

Two timeouts govern when a head block closes:

• trace_idle_period (default 10 s) closes a head block when no new span has arrived for that trace ID. Most traces close this way.
• max_block_duration (default 30 m) forces a flush of the oldest head block when the cap is reached. This prevents long-lived traces from occupying memory indefinitely.

The WAL is the safety net. Every time a head block is updated, the WAL records the change. On process restart the WAL is replayed into in-memory state, and the trace-by-id lookup continues to work until the block is flushed.

How to configure it

A production ingester config pins the lifecycle, the WAL, and the flush cadence:

ingester:
  # Close a head block after this period of no activity on a trace.
  trace_idle_period: 10s

  # Force-flush a head block after this absolute age regardless of activity.
  max_block_duration: 30m

  # How often the head block is fsynced to the WAL.
  flush_check_period: 5s

  # How often completed blocks are pushed to object storage.
  # Defaults to max_block_duration. Lower values produce smaller blocks
  # more often; higher values produce larger blocks less often.
  # The compactor merges small blocks later.
  max_block_bytes: 524288000   # 500 MiB; flush a block at this size too

  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3
      heartbeat_timeout: 1m
    heartbeat_period: 5s
    join_after: 10s
    observe_period: 10s
    final_sleep: 0s

  # The WAL lives here. NVMe-backed local SSD is the right answer.
  # Network-attached storage does not have the IOPS profile the WAL
  # needs under write load.

Three production details to call out:

• replication_factor: 3 requires at least three ingester pods. Two is not enough for the write quorum to succeed.
• trace_idle_period and max_block_duration interact. A trace that emits spans every five seconds never closes via idle and waits for max_block_duration. A trace that emits spans every thirty seconds closes via idle in 10 s.
• max_block_bytes caps a single block. Larger blocks reduce the number of blocks the compactor must merge but increase the blast radius of a single block loss.

How to validate it

Six checks confirm the ingester is doing its job:

1. Confirm the ingester is ready and registered in the ring:

curl -s http://tempo.internal:3200/ingester/ready
# ready
curl -s http://tempo.internal:3200/ingester/ring | jq .
# {
#   "name": "ingester",
#   "tokens": [...],
#   "members": [
#     {"addr": "tempo-0:3200", "state": "ACTIVE"},
#     {"addr": "tempo-1:3200", "state": "ACTIVE"},
#     {"addr": "tempo-2:3200", "state": "ACTIVE"}
#   ]
# }

2. Confirm the WAL is being checkpointed. A healthy cluster keeps the last checkpoint within minutes, not hours:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_ingester_local_checkpoint_manager_last_saved_timestamp
# tempo_ingester_local_checkpoint_manager_last_saved_timestamp  1723655400
# That value should be within (now - max_block_duration) seconds.

3. Confirm blocks are flushing. The local block counter should trend up under load and back down after a flush:

curl -s http://tempo.internal:3200/metrics \
  | grep -E '^tempo_ingester_local_blocks\b'
# tempo_ingester_local_blocks  42

4. Confirm flushes are succeeding. The failed-flush counter should be zero under healthy operation:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_ingester_failed_flushes_total
# (no output means zero)

5. Confirm the trace-by-id lookup works against a freshly accepted trace. Send a span, then read it back:

# Send
otel-cli span export --endpoint tempo.internal:4317 \
  --service checkout --name POST /charge
TRACE_ID=$(otel-cli span ls --limit 1 --format json | jq -r '.[0].TraceId')

# Wait briefly for the head block to be written
sleep 2

# Read by trace ID
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | jq '.batches | length'
# 1

6. Confirm the trace survives a single ingester restart. With replication_factor: 3 the trace must be readable from one of the other two pods. The tempo_ingester_trace_forgotten counter should remain at zero.

How it can fail

Six shapes appear repeatedly:

1. WAL grows without flushing. Object storage credentials have rotated, or the bucket is throttling, or the bucket is unreachable. The ingester keeps accepting spans, the WAL grows, the disk fills. Symptom is tempo_ingester_failed_flushes_total rising alongside disk pressure.
2. WAL corruption on disk. A power loss or a kernel bug corrupts the WAL file. The ingester cannot replay on restart and drops the affected blocks. Symptom is tempo_ingester_wal_corruptions_total rising.
3. Ring instability. A misconfigured join_after or heartbeat_timeout causes the ingester to flap between ACTIVE and JOINING. The distributor sees different ring contents each second; spans land on the wrong pod and return 503. Symptom is tempo_ingester_lifecycler_ring_inconsistencies_total rising.
4. Memory exhaustion from long-lived traces. A single trace that emits spans for hours stays in a head block. A burst of such traces fills the ingester heap. Symptom is go_memstats_heap_inuse_bytes growing and the OOM killer firing.
5. Block size cap too small. max_block_bytes set below normal trace size produces many small blocks. The compactor cannot keep up; the bucket ends up with millions of blocks. Symptom is tempo_compactor_blocks_compacted_total flat while tempo_querier_blocks_scanned_total rises.
6. trace_idle_period too short. A trace that legitimately has gaps of more than trace_idle_period closes its head block before all spans arrive. Subsequent spans for the same trace ID create a new head block. The querier returns two halves of one trace. Symptom is user complaints about truncated traces.

How to troubleshoot it

The diagnostic order:

1. Is the ingester in the ring? Check tempo_ingester_lifecycler_is_ready. A value of 0 means the ingester has not registered and the distributor cannot route to it.
2. Is the WAL draining? Check tempo_ingester_local_checkpoint_manager_last_saved_timestamp. A timestamp older than max_block_duration means flushes are not completing.
3. Are flushes failing? Check tempo_ingester_failed_flushes_total broken down by reason. A non-zero value means the bucket is not accepting writes.
4. Is the disk filling? Check df -h on the WAL path. The WAL is sized to hold at most a few minutes of spans at peak; a full WAL means flushes have been failing for tens of minutes.
5. Is memory under pressure? Check the Go runtime metrics go_memstats_heap_inuse_bytes and go_memstats_heap_alloc_bytes. A heap that grows past the configured limit triggers the OOM killer.
6. Are traces readable? Hit /api/traces/{id} for a known trace. If the trace is in the WAL but not in the bucket, the problem is the flush path. If it is in the bucket but not readable, the problem is the querier.

Security implications

The ingester is local-network-only:

• WAL on local disk. The WAL contains span payloads. Anyone with read access to the WAL path can read every span that arrived in the last flush window. Restrict the WAL path to root or to a dedicated service user.
• No inbound network. The ingester accepts traffic from the distributor over the in-cluster network. The distributor must not be exposed publicly; the ingester inherits that protection.
• Tracing sensitive data. If traces contain PII or secrets, the WAL and the bucket both contain them. Tempo does not redact at ingest; redaction is the responsibility of the SDK or the collector.

Performance implications

The ingester is the most expensive component to run:

• Memory. Each open head block holds all spans for one trace ID. A burst of long-lived traces can exhaust heap memory.
• Disk. The WAL writes one record per span. A 20 MiB/s ingest writes roughly 20 MiB/s to the WAL until the next checkpoint.
• Network. Each span is replicated replication_factor times inside the ring (the distributor fans out). At replication_factor: 3 a 20 MiB/s ingest becomes a 60 MiB/s internal stream.
• CPU. Span decoding and WAL encoding consume CPU. A sustained 30 MiB/s ingest saturates one CPU core per ingester pod.

Production guidance

• Pin the WAL to a fast, local disk. NVMe SSD is the right answer. Network-attached storage does not have the IOPS profile.
• Run at least three ingester pods so replication_factor: 3 has a quorum.
• Set trace_idle_period long enough that legitimate long- running traces close naturally. 10 s is conservative; some teams run 30 s.
• Alert on tempo_ingester_local_checkpoint_manager_last_saved_timestamp age. A stale checkpoint is the earliest signal of a flush problem.

Verification

You should now be able to answer:

• What is the trace lifecycle from first span to flushed block?
• What is the role of the WAL and how is it truncated?
• What happens when trace_idle_period is too short?
• What is the meaning of tempo_ingester_local_checkpoint_manager_last_saved_timestamp?
• Why must the WAL be on local SSD rather than network storage?

Quiz