Tempo Compactor Scaling

At the end of the month the storage bill was 4.2x the previous month’s. The team had not changed retention or ingest. The compactor pod had been silently failing for ten days after a Helm upgrade moved the bucket credentials into a Kubernetes Secret that the compactor’s service account could not read. Blocks accumulated without merging. The bucket grew past ten million objects. The bill arrived first; query latency rose three days later.

This lesson is the discipline of running the Tempo compactor: why it is a singleton, what its cadence costs, and the failure shapes that show up when it stops.

What it is

The Tempo compactor walks the bucket, merges small blocks into larger ones, and (in v2) deduplicates blocks that share trace IDs. It is mostly stateless but operates on the entire bucket, which is why the default deployment is a singleton.

The compactor is the silent keeper of two operational metrics:

• Block count. The number of objects in the bucket. The dominant driver of query latency.
• Bucket size. The bytes in the bucket. The dominant driver of the storage bill.

When the compactor is healthy, both metrics grow slowly. When it is failing or stopped, both grow faster than ingest.

Why a sysadmin cares

Three operational pains are specific to the compactor:

1. Bucket growth. A stopped compactor produces a bucket that grows monotonically. After a week the storage bill reflects weeks of data, not days.
2. Query latency. The querier scans blocks per query. A bucket with millions of objects spends most of its query time in ListObjectsV2 calls, not in fetching block contents. Query p99 latency rises with block count.
3. Block-format drift. A Tempo upgrade that changes the block format requires a compactor that understands the new format. A pinned-old chart on a new binary produces blocks the compactor cannot merge.

The compactor is mandatory in microservices mode. The binary tolerates its absence; the bucket does not.

How it works

The compactor runs a per-tenant compaction loop. For each tenant, it:

1. Lists the tenant’s blocks in the bucket.
2. Groups blocks by their compaction window (default 1h).
3. Pairs the small blocks within each window.
4. Reads both blocks, merges them into a new block, writes the new block back to the bucket.
5. Marks the source blocks for deletion.
6. Records the deletion in the markers file so the bucket lifecycle policy can reap the bytes.

  S3 / GCS / Azure
        |
        |  list blocks per tenant
        v
  +--------------------------------------+
  | Compactor (singleton by default)     |
  |                                      |
  |   for each tenant:                   |
  |     for each compaction_window:      |
  |       pair small blocks              |
  |         read both                    |
  |         merge into new block         |
  |         write new block              |
  |         mark sources for deletion    |
  |         record marker                |
  +--------------------------------------+
        |
        v
  Bucket lifecycle policy reaps deleted bytes

In v2 the compactor also deduplicates blocks that share trace IDs. A trace that arrived on two ingesters (because the ring rebalanced mid-trace) produces two partial blocks; the compactor merges them into a single complete block.

How to configure it

A production compactor config pins the cadence and the retention. The values below match the production defaults for a mid-scale cluster:

compactor:
  compaction:
    block_retention: 48h
    compaction_window: 1h
    max_compaction_objects: 1000000
    max_block_bytes: 107374182400   # 100 GiB per merged block
    flush_interval: 15m
    compacted_block_timeout: 30m

  ring:
    kvstore:
      store: memberlist

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}

Three details to call out:

• block_retention: 48h deletes blocks older than two days. The compactor marks them; the bucket lifecycle policy reaps the bytes. Both must be configured.
• compaction_window: 1h means the compactor merges all blocks inside each one-hour window into one block. A larger window produces larger merged blocks; a smaller window produces more blocks per cycle. The default is a balance.
• max_compaction_objects: 1000000 caps the number of objects the compactor will read in one cycle. The cap prevents a runaway cycle from saturating the compactor pod.

When the bucket is sharded, the additional values:

compactor:
  ring:
    kvstore:
      store: memberlist
  sharding_ring:
    enabled: true
    kvstore:
      store: memberlist

Severity: CONFIGURATION. Reloading the compactor config requires a process restart.

How to validate it

Severity: READ-ONLY.

1. Confirm the compactor is ready and registered:

curl -s http://tempo-compactor:3200/ready
# ready

2. Confirm compaction is making progress:

curl -s http://tempo-compactor:3200/metrics \
  | grep -E '^tempo_compactor_(blocks_compacted|blocks_marked)_total'
# tempo_compactor_blocks_compacted_total 4231
# tempo_compactor_blocks_marked_total   4198

The two counters should track each other within a small multiple. A flat blocks_compacted counter for hours means the compactor is wedged.

3. Confirm the bucket has a sane block count:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix 'blocks/' \
  --max-items 0 \
  --query 'Length' \
  | jq .
# 482301

A block count growing faster than ingest rate per hour means the compactor is not keeping up.

4. Confirm the compactor can reach the bucket:

curl -s http://tempo-compactor:3200/metrics \
  | grep tempo_compactor_storage
# tempo_compactor_storage_total_bytes 2.3e11

A missing or zero counter means the bucket is unreachable.

How it can fail

Six shapes appear repeatedly:

1. Bucket credentials expired or rotated. The compactor cannot list blocks. Symptom is the compactor logs NoCredentialProviders or AccessDenied and the blocks_compacted_total counter is flat.
2. Compactor missing after a Helm upgrade. The new chart does not deploy a compactor by default in some configurations. Symptom is kubectl get statefulset showing no compactor pod and the bucket growing.
3. Compaction window too small for the workload. A 10-minute window with 50k spans/sec produces too many small blocks for the compactor to merge in one cycle. Symptom is the compactor using 100% CPU but blocks_compacted_total advancing slowly.
4. max_compaction_objects too low. The compactor pauses each cycle after reading the cap, then resumes on the next cycle. Symptom is the cycle metric spiking and the blocks_compacted_total advancing in step-wise jumps.
5. Block-format drift after a binary upgrade. A new Tempo binary writes blocks in a new format. An old compactor cannot read them. Symptom is the compactor logs unknown block format and skips blocks. A pin mismatch.
6. Compactor pod OOM-killed on a backlog. A compactor that has been offline for a week tries to read millions of blocks in the first cycle. Symptom is kubectl describe pod showing OOMKilled and the compactor pod restarting repeatedly.

How to troubleshoot it

The diagnostic order:

1. Is the compactor pod running? kubectl get pod -l app=tempo-compactor. A missing pod is a deployment bug.
2. Can the compactor reach the bucket? Check the compactor logs for credential errors and the bucket policy for the compactor’s IAM role.
3. Is the compactor making progress? Check tempo_compactor_blocks_compacted_total. A flat counter for more than one cycle means something is wedged.
4. Is the compactor OOM-killed? Check kubectl describe pod for OOMKilled events.
5. Is the bucket growing faster than the compactor can drain? List the bucket and divide by the rate of blocks_marked_total.
6. Are blocks the right format? Check the compactor logs for unknown block format lines. A pin mismatch between binary versions.

Security implications

The compactor has two attack surfaces:

• Bucket credentials. The compactor reads every block in the bucket. A leaked AWS key with s3:GetObject and s3:ListBucket on the Tempo bucket is a trace-data breach. Use scoped credentials and rotate them.
• Compactor HTTP endpoint. The /ready and /metrics endpoints expose operational state. Bind the HTTP listen address to a private interface; do not expose the compactor port externally.

Performance implications

The compactor is CPU- and network-bound during the compaction cycle. The cost is bounded by the cycle length and the max_compaction_objects cap. Three knobs:

• Cycle length. Default 15 minutes. A shorter cycle produces smaller merges and finishes sooner; a longer cycle produces larger merges and finishes later.
• Compaction window. Default 1 hour. A larger window merges more blocks per cycle and produces fewer merged blocks in the bucket. The trade-off is per-block read cost.
• Max compaction objects. Default 1,000,000. The cap prevents a single cycle from saturating the compactor pod.

For most platforms a single compactor pod with 4 vCPU and 8 GiB is sufficient for a bucket of up to ~10 million blocks per cycle.

Production guidance

• Run exactly one compactor by default. Only enable sharded compaction when a single compactor cannot keep up with the bucket.
• Set block_retention to the longest time you will ever need to query, and configure the bucket lifecycle to match.
• Alert on tempo_compactor_blocks_compacted_total advancing and on the bucket object count.
• Pin the compactor binary to the same version as the rest of the cluster. A version mismatch produces block-format errors.

Verification

You should now be able to answer:

• Why does the compactor run as a singleton by default?
• What does compaction_window control, and what is the trade-off of changing it?
• What is the cost of a stopped compactor over hours, days, and weeks?
• How do you validate that the compactor is making progress?
• When is sharded compaction worth the operational cost?

Quiz