Tempo Rotation Behavior

The on-call engineer opens the Tempo dashboard on Thursday morning. Block count has doubled in the last week. The compactor’s blocks_compacted_total is climbing slowly; the ingester’s blocks_created_total is climbing fast. The compactor cannot keep up with the new block creation rate. Block retention is at the documented value; compaction concurrency is at the documented value. Something in the ingest rate has crossed a threshold the compaction settings were not sized for.

This is the Tempo rotation behaviour that production teams hit when the ingest rate climbs. The rotation cycle is a closed loop: ingester creates blocks, compactor merges them, object storage holds them, compactor evicts on retention. When the ingest rate climbs, the loop opens.

What Tempo block rotation is

Tempo block rotation is the closed loop that moves spans from the ingester’s in-memory trace buffer to long-term object storage, and from long-term storage to eviction. The loop has four stages:

1. Block creation. The ingester writes a trace’s spans to an in-memory block. When the block reaches a configured size (max_block_size_bytes) or age (max_block_duration), the ingester flushes the block to object storage.
2. Block listing. The compactor periodically lists the blocks in object storage. It groups them by tenant and time window.
3. Compaction. The compactor merges adjacent blocks in the same time window into a single block. Compaction reduces the block count for the same trace volume, which reduces the per-query I/O cost and the per-tenant index size.
4. Retention. The compactor evicts blocks older than compactor.compaction.block_retention. Eviction frees object storage and respects the documented retention window.

The four stages form a closed loop. The loop is healthy when the compactor’s merge rate equals the ingester’s flush rate divided by the compaction factor. The loop is unhealthy when the ingester’s flush rate exceeds the compactor’s processing rate; the backlog grows; the per-tenant block count climbs; the per-query I/O cost climbs with it.

Why a sysadmin cares

A Tempo rotation backlog degrades the tracing platform. The ingester’s in-memory queue grows; the per-query I/O cost grows; the block listing grows; the compactor falls further behind. Eventually the ingester cannot flush blocks because the object storage is saturated. The distributor back- pressures. The platform goes silent.

The cost of the rotation incident is paid in the next two hours of the on-call engineer’s night. The cost of the next rotation incident is paid by everyone who keeps raising the ingest rate without raising the compaction settings.

How it works

Tempo’s rotation loop is observable through four metric families: ingester block creation, compactor compaction, object storage size, and block retention.

Ingester  --flush-->  Object storage  --list-->  Compactor
   |                       |                       |
   |                       |                       +-- merge adjacent
   |                       |                       +-- evict on retention
   |                       |
   +-- size or age bound   +-- tenant-scoped        +-- compaction concurrency
       (max_block_*)           (block-format)

The closed loop:

ingester creates block
   |
   v
object storage holds block
   |
   v
compactor lists blocks
   |
   +-- if adjacent blocks exist: merge
   |
   +-- if block older than retention: evict
   |
   v
block count falls

The open loop (rotation backlog):

ingester creates block (rate R)
   |
   v
object storage holds block (rate R)
   |
   v
compactor lists blocks (rate R / compaction_factor)
   |
   v
compactor merges blocks (rate R / compaction_factor / concurrency)
   |
   v
block count grows at (R - R / compaction_factor / concurrency)

The backlog grows when R / compaction_factor / concurrency is less than R. In production, this is observable as a climb in tempo_ingester_blocks_created_total that outpaces tempo_compactor_blocks_compacted_total.

Under the hood

How to configure it

The fix for a rotation backlog is to tune the compactor’s concurrency and the ingester’s flush bounds. The prevention is a baseline of expected per-tenant block count.

# /etc/tempo/tempo.yaml  (Tempo ingester)
ingester:
  # Block flush bounds. A higher max_block_size reduces the
  # number of small blocks; a higher max_block_duration
  # delays flushing and reduces write throughput.
  max_block_size_bytes: 5242880   # 5 MiB; default
  max_block_duration: 30m         # default
  trace_idle_period: 10m          # how long an unflushed trace lives

# /etc/tempo/tempo.yaml  (Tempo compactor)
compactor:
  compaction:
    # Block retention window. Blocks older than this are
    # evicted by the compactor.
    block_retention: 168h   # 7 days; default for production

  # Compactor concurrency. Each concurrent worker merges one
  # block pair at a time. Raise this to raise the merge
  # throughput.
  concurrency: 2   # default; raise to 4 or 8 for high ingest

# /etc/tempo/tempo.yaml  (Tempo distributor)
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
  # The distributor's rate limit bounds the maximum ingest.
  # Lowering it during an incident eases the rotation loop.
  rate_limit: 100000

The diagnostic discipline: every rotation parameter should be readable from a single source of truth. A rotation parameter that was tuned for the deploy-time traffic is wrong six months later when the traffic has doubled.

How to validate it

When block count is climbing, the first read is the ingester block creation rate, the compactor compaction rate, and the per-tenant block count.

# READ-ONLY: ingester block creation rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_blocks_created_total'

# READ-ONLY: compactor compaction rate.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_compactor_blocks_compacted_total'

# READ-ONLY: blocks marked for compaction (the backlog).
curl -s http://tempo:3200/metrics \
  | grep '^tempo_compactor_blocks_marked_for_compaction'

Illustrative output during a rotation backlog:

tempo_ingester_blocks_created_total 1.85e+06
   # ingester is creating 1850000 blocks over the lifetime

tempo_compactor_blocks_compacted_total 9.2e+05
   # compactor has compacted 920000 blocks; ingester has
   # created roughly twice that; the backlog is ~630000
   # blocks

tempo_compactor_blocks_marked_for_compaction 630000
   # the canonical backlog indicator

If the ingester creation rate outpaces the compactor rate, the compactor is undersized. Raise the compactor’s concurrency and revalidate.

# READ-ONLY: per-tenant block count.
# Tempo exposes a generated metric for blocks per tenant.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_blocks_per_tenant'

A per-tenant block count that has doubled compared to the baseline is the textbook symptom. The fix is at the compactor; the symptom is at the tenant.

How it can fail

The six failure shapes that account for the great majority of Tempo rotation backlogs:

1. Compactor concurrency undersized. The compactor’s concurrency was set to 2 at deploy time; the traffic has doubled; the compactor cannot keep up. The backlog grows. Symptom: blocks_marked_for_compaction climbs.
2. block_retention shorter than the time-window grouping. The compactor evicts blocks faster than it can merge them; the platform loses trace coverage before the compactor can compact adjacent blocks.
3. max_block_duration too short for the ingest rate. The ingester flushes many small blocks on age. The compactor has to merge more blocks per unit time. The per-block overhead dominates the compaction throughput.
4. Object storage throughput saturated. The object storage backend cannot sustain the write throughput of the ingester flushes. Flushes queue at the ingester; the in-memory block queue grows.
5. Compactor is not running. The compactor component is not deployed, or has crashed. The ingester flushes blocks; no compaction runs; the backlog grows monotonically.
6. Tenant cardinality explosion. A new tenant was on-boarded with a per-tenant block format. The compactor groups by tenant; the per-tenant block count explodes; the compactor’s per-tenant merge throughput does not scale.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the bottleneck, then act.

1. Confirm the symptom. Read tempo_compactor_blocks_marked_for_compaction. A climb is the canonical backlog indicator.
2. Locate the bottleneck. Compare tempo_ingester_blocks_created_total to tempo_compactor_blocks_compacted_total. If the ingester rate outpaces the compactor rate, the compactor is the bottleneck. If the compactor rate is flat while the ingester rate climbs, the compactor is not running.
3. Locate the object storage. Read the object storage backend’s metrics. If the write throughput is saturated, the bottleneck is downstream of the compactor.
4. Form a hypothesis. Identify when the backlog started growing. The hypothesis is almost always correlated with an ingest rate change in the change log.
5. Find evidence. Cross-reference with the deploy log, the sampling config, and the namespace on-boarding log. The culprit is almost always correlated with one of those.
6. Act. Raise the compactor’s concurrency. Validate that the compactor rate now keeps pace with the ingester rate. Do not add storage until the backlog is closed.

Security implications

A Tempo rotation backlog is observable through the metrics endpoint. If the metrics endpoint is exposed without authentication, an attacker can infer the platform’s ingest rate, retention window, and per-tenant block count. The fix is a documented metrics endpoint configuration that does not expose these signals to untrusted clients.

Block retention that is shorter than the audit window causes the platform to lose trace coverage that audit requires. The fix is a documented retention window that meets the audit requirement, regardless of the ingest rate.

Performance implications

A Tempo rotation backlog degrades per-query I/O cost. A platform with 1000 blocks per tenant and a compaction factor of 5 has 200 effective blocks per query. A platform with 10000 blocks per tenant and the same compaction factor has 2000 effective blocks per query. The per-query cost is roughly proportional to the effective block count.

A rotation backlog also degrades the compactor’s per-tenant merge throughput. A compactor that is processing 1000 blocks per minute on a healthy platform is processing 500 blocks per minute on a backlogged platform, because the compactor spends time listing blocks it cannot merge.

Production guidance

• Track tempo_compactor_blocks_marked_for_compaction per compactor instance. Alert before the backlog exceeds the per-tenant baseline by a factor of two.
• Right-size the compactor’s concurrency from the documented per-tenant block count. A concurrency of 2 is sufficient for 1000 blocks per minute; raise to 4 or 8 for 4000 blocks per minute.
• Pin the compactor’s block_retention to the documented audit window. Do not lower retention to ease backlogs.
• Run the compactor as a separate component with its own resource budget. Co-locating the compactor with the ingester causes resource contention during high ingest.
• Document a baseline for every rotation parameter. The baseline is the source of truth for the next incident.

Verification

• What are the four stages of the Tempo rotation loop, and what is the canonical backlog indicator?
• What is the diagnostic order when tempo_compactor_blocks_marked_for_compaction is climbing?
• What is the most common cause of a Tempo rotation backlog?
• Where in the Tempo configuration is the compactor’s concurrency set, and what is the metric that confirms the compactor is keeping pace with the ingester?

Quiz