The Compactor

A querier takes forty seconds to return a TraceQL result that should take one. The on-call engineer opens the bucket metrics and sees three hundred thousand small blocks. The compactor stopped running three weeks ago, the ingester kept flushing, and the querier now scans every block in the bucket to answer every query. A team restores the compactor; the queue clears in two hours.

This lesson describes the Tempo compactor: why it exists, how it walks the bucket, and what it costs to run.

What it is

The Tempo compactor is a stateless singleton service. Its job is to walk the trace bucket, merge small blocks into larger ones, and delete blocks older than the configured retention. It is the LSM-tree “compact” stage of Tempo’s storage path.

In default configuration, the compactor runs alone. Two compactors racing on the same block can corrupt it, so the service uses a KV-lease to elect a single leader. Sharded compaction is supported but requires explicit configuration.

Why a sysadmin cares

Three operational pains are specific to the compactor:

1. Unbounded block growth. The ingester flushes every max_block_duration per tenant. Without the compactor, the block count grows linearly with time. The querier scans every block that intersects a query window; the cost grows with the block count, not the data size.
2. Retention enforcement. The compactor is the component that deletes blocks. Tempo deletes by removing the index entry from the block metadata; the bytes are left to the bucket’s lifecycle policy. A misconfigured compactor either keeps too much or deletes too little.
3. Trace ID dedup in v2 blocks. Tempo v2 blocks rewrite the data so that traces with shared IDs across multiple blocks are deduplicated. The compactor performs the rewrite; the querier reads the deduped form.

How it works

The compactor runs two background loops on the bucket:

  Bucket
    |
    +-- loop A: scan blocks by tenant + window
    |     |
    |     v
    |   list blocks older than the compaction window
    |     |
    |     v
    |   group by (tenant, window)
    |     |
    |     v
    |   merge into one output block
    |     |
    |     v
    |   write to bucket, mark originals for deletion
    |
    +-- loop B: scan blocks by tenant + window
          |
          v
        list blocks older than retention
          |
          v
        delete block metadata (bucket lifecycle deletes bytes)

Two loops because the operations have different costs and cadences:

• Compaction loop. Runs continuously. Walks the bucket, finds small blocks within the compaction window (default 1 hour), and merges them into a single block per tenant per window. A block that is already larger than compaction.compaction_block_size_bytes is left alone.
• Retention loop. Runs every compactor.compaction.retention_check_duration (default 1 h). Walks the bucket, finds blocks older than block_retention, and marks them for deletion. The bucket lifecycle policy deletes the bytes after a configurable grace period.

How to configure it

A production compactor config pins the compaction window, the retention, and the work directory:

compactor:
  compaction:
    # Compaction windows are 1 hour. Smaller windows mean more
    # frequent compaction; larger windows mean more blocks per
    # compactor pass.
    compaction_window: 1h

    # Max size of a single compacted block. Blocks larger than
    # this are not merged with their neighbours.
    compaction_block_size_bytes: 536870912   # 512 MiB

    # Retention window. Blocks older than this are marked for
    # deletion. Pair this with the S3 bucket lifecycle policy.
    block_retention: 168h   # 7 days

    # How often the retention loop runs.
    retention_check_duration: 1h

    # Local work directory. Must be sized to hold the largest
    # compacted block plus overhead. NVMe SSD is the right answer.
    working_directory: /var/tempo/compactor

    # How many blocks to compact in parallel per tenant.
    max_compaction_objects: 100

Three production details to call out:

• compaction_window: 1h is the right default. Smaller windows produce smaller compactions more often; larger windows produce larger compactions less often and increase the local disk needed.
• block_retention and the bucket lifecycle policy must agree. If block_retention: 168h and the lifecycle expires objects after 24 hours, the compactor marks blocks for deletion but the bytes vanish before a recovery could restore them.
• working_directory must be on local disk with enough free space for the largest merged block. A 512 MiB compacted block needs at least 1 GiB of free space during the merge.

How to validate it

Five checks confirm the compactor is doing its job:

1. Confirm the compactor is ready and elected leader:

curl -s http://tempo.internal:3200/compactor/ready
# ready
curl -s http://tempo.internal:3200/compactor/ring | jq .
# {
#   "name": "compactor",
#   "members": [{"addr": "tempo-compactor-0:3200", "state": "ACTIVE"}]
# }

2. Confirm compaction is progressing. The tempo_compactor_blocks_compacted_total counter should rise over time:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_compactor_blocks_compacted_total
# tempo_compactor_blocks_compacted_total  1284

3. Confirm the block count is bounded. Use aws s3api to count blocks in the bucket:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix 'blocks/ingester/' \
  --output json | jq '.KeyCount'
# 4721

A healthy cluster holds tens of thousands of blocks; a broken compactor produces hundreds of thousands.

4. Confirm the retention loop is deleting old blocks. The tempo_compactor_blocks_marked_for_deletion_total counter should rise as the retention window advances:

curl -s http://tempo.internal:3200/metrics \
  | grep tempo_compactor_blocks_marked_for_deletion_total
# tempo_compactor_blocks_marked_for_deletion_total  384

5. Confirm a query that previously scanned many blocks now scans few. The tempo_querier_blocks_scanned_per_query histogram should trend down after the compactor catches up:

curl -s http://tempo.internal:3200/metrics \
  | grep 'tempo_querier_blocks_scanned_per_query_bucket' | head
# tempo_querier_blocks_scanned_per_query_bucket{le="1"} 412
# tempo_querier_blocks_scanned_per_query_bucket{le="10"} 891

How it can fail

Five shapes appear repeatedly:

1. Compactor is not running. Block count grows without bound. Querier latency climbs as the bucket grows. Symptom is tempo_compactor_blocks_compacted_total flat at zero and bucket object count rising without limit.
2. Two compactors racing. A second compactor pod starts without realising the first is the leader. Both compact the same block; one overwrites the other’s output. Symptom is corrupted blocks appearing in the bucket; the querier logs parquet: invalid footer errors.
3. Bucket throttling. The compactor downloads a hundred blocks at once. S3 throttles with SlowDown. Symptom is tempo_compactor_failed_compactions_total rising alongside 503 errors in the compactor logs.
4. Local disk full. The working directory cannot hold the next merged block. Symptom is tempo_compactor_disk_out_of_space_total rising; compactions fail.
5. Retention loop not deleting. The block_retention window is too long, or the bucket lifecycle is misconfigured. Symptom is the bucket size growing past the planned capacity; cost increases silently.

How to troubleshoot it

The diagnostic order:

1. Is the compactor running? Check pod status. A crashed compactor is the obvious case; a running compactor that has not registered in the ring is the less obvious one.
2. Is the compactor the leader? Check tempo_compactor_ring_members. More than one ACTIVE member means a race; pick one and stop the others.
3. Are compactions succeeding? Check tempo_compactor_blocks_compacted_total. A flat counter despite a running compactor means the compaction loop is failing.
4. Is the bucket throttling? Check the compactor logs for SlowDown from S3. If present, raise the throttling retry budget or back off the parallel compaction count.
5. Is the working directory sized correctly? Check df -h on working_directory. A full disk means compactions cannot write output blocks.
6. Is the retention loop deleting blocks? Check tempo_compactor_blocks_marked_for_deletion_total. A flat counter despite old blocks in the bucket means the retention loop is not running or block_retention is wrong.

Security implications

The compactor is local-network-only:

• Bucket credentials. The compactor must write merged blocks to the bucket. The same scoped credentials used by the ingester apply.
• Local work directory. The merged blocks live briefly on local disk. Anyone with read access to the working directory can read span payloads. Restrict the path to root or to a dedicated service user.
• No inbound network. The compactor accepts only the /ready and /metrics endpoints on its local listen port. No inbound data traffic.

Performance implications

The compactor is CPU- and network-bound during the compaction window:

• CPU. Block decoding, merging, and encoding consume CPU proportional to block size. A 1 GiB compaction consumes roughly one CPU minute on a modern x86.
• Network. The compactor downloads input blocks and uploads output blocks. At 100 MiB/s sustained, the compactor saturates the local link.
• Local disk. The working directory holds the largest merged block. A 512 MiB compacted block needs at least 1 GiB of free space.
• Cost. The compactor performs one compaction per tenant per window per cycle. A cluster with 200 tenants and a 1-hour window performs 200 compactions per hour.

Production guidance

• Run exactly one compactor pod. More than one is a race; less than one is a single point of failure for retention.
• Size the working directory to twice the largest merged block. NVMe SSD is the right answer.
• Set block_retention to the longest time you will ever need to query, and configure the bucket lifecycle to match.
• Alert on tempo_compactor_blocks_compacted_total rate. A falling rate means the compactor cannot keep up.

Verification

You should now be able to answer:

• What does the compactor do, and why must it run as a singleton?
• What is the role of compaction_window vs block_retention?
• How does the compactor delete a block?
• Why must the working directory be on local SSD rather than network storage?
• What is the operational cost of the compactor not running?

Quiz