Tempo Retention

A platform team inherits a Tempo instance whose bucket has been growing for 14 months. The block_retention setting is the default — 0s, meaning retain forever. The bucket holds 280 TB. The on-call budget meeting assigns the trace store a 90-day retention target. The team sets block_retention: 2160h. The bucket does not shrink. The investigation reveals that the compactor process has not run since the cluster was upgraded three months prior; the ingester is still writing new blocks, but no block has ever been deleted. The fix is to start the compactor and let it delete the now-eligible blocks.

Retention in Tempo is enforced by the compactor. Without a running compactor, retention is theoretical.

What it is

Tempo retention is the policy that decides how long a trace block lives in object storage. The policy has three parts:

1. Global retention. compactor.compaction.block_retention in tempo.yaml. The default is 0s (forever). A non-zero duration marks every block older than the duration as eligible for deletion.
2. Per-tenant overrides. An overrides.yaml file referenced from the main config lets specific tenants have longer or shorter retention.
3. Compactor execution. The compactor process is the only component that performs the deletion. With no compactor, no block is ever deleted regardless of the policy.

The compactor also merges small blocks into larger ones to reduce listing cost and query latency. This is a separate operation but the same process.

Why a sysadmin cares

Trace data grows faster than metrics or logs. At 50k spans/sec with 4 KB per span compressed, the daily volume is around 17 TB. Without a retention policy, the bucket grows by ~500 TB/month. The cost is not abstract: a 1 PB bucket on S3 Standard-IA is roughly $13k/month in storage alone, plus egress on every query.

A retention policy is also an investigation depth decision. Three days is enough for most active incidents; a 90-day policy is enough for slow-burn post-mortems; a one-year policy supports quarterly reviews. The trade-off is explicit: cost vs depth.

How it works

   ingester  ---->  s3://bucket/blocks/...
                          |
                          v
                   compactor (cron)
                          |
                  +-------+-------+
                  |               |
                  v               v
             list blocks     compact eligible
             older than        (merge small
             block_retention   blocks into
                  |            fewer, larger)
                  v               |
             delete             upload merged
             original           blocks

The compactor lifecycle

The compactor runs a periodic loop, configurable with compactor.compaction.compaction_interval (default 8 hours, but check release notes for current default). On each cycle:

1. List blocks under the configured prefix.
2. For each block, check whether the block’s max time is older than now - block_retention. If so, mark for deletion.
3. For each pair of overlapping blocks in the same tenant, merge them into one larger block (subject to max_block_bytes and other limits).
4. Delete the originals after the merged block is durable.

Overrides

Per-tenant overrides are loaded from an external YAML file:

# overrides.yaml
overrides:
  tenant-a:
    block_retention: 720h    # 30 days
  tenant-b:
    block_retention: 2160h   # 90 days

The Tempo main config references this file:

overrides:
  per_tenant_override_config: /etc/tempo/overrides.yaml

A tenant whose ID is not in the file inherits the global block_retention. A tenant with an explicit override gets the override.

Under the hood

How to configure it

Global retention

# /etc/tempo/tempo.yaml
compactor:
  compaction:
    block_retention: 2160h          # 90 days, the safe default
    compaction_interval: 8h
    max_block_bytes: 100_000_000_000   # 100 GB
    retention_concurrency: 10

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      prefix: blocks

Severity: DATA-LOSS-RISK. Setting block_retention deletes data older than the value. Validate the value, the bucket, and the prefix before applying.

The compactor must be enabled with target: compactor in microservices mode (the default if running with the Helm chart).

Per-tenant overrides

# /etc/tempo/tempo.yaml
overrides:
  per_tenant_override_config: /etc/tempo/overrides.yaml
  defaults:
    block_retention: 168h          # 7 days for tenants without an override

# /etc/tempo/overrides.yaml
overrides:
  acme-prod:
    block_retention: 720h          # 30 days for the paying tenant
  acme-archive:
    block_retention: 2160h         # 90 days for the archive tenant
  free-tier:
    block_retention: 24h           # 24 hours for free-tier tenants

Severity: DATA-LOSS-RISK per tenant. A typo in the tenant ID silently falls back to the default; validate after applying.

How to validate it

Severity: READ-ONLY.

# 1. Confirm the compactor is running and registered
curl -s http://tempo-compactor:3200/ready

# 2. Confirm the active config
curl -s http://tempo-compactor:3200/api/status | jq .

# 3. Inspect the per-tenant override via the runtime config endpoint
curl -s http://tempo-compactor:3200/config | jq .

# 4. Confirm blocks older than block_retention are being deleted
aws s3 ls s3://tempo-traces-prod/blocks/single-tenant/1/ \
  --recursive | awk '{print $4}' \
  | grep -E '^[0-9a-f-]+/$' | head

Real output:

$ curl -s http://tempo-compactor:3200/api/status | jq .
{
  "compactor": {
    "healthy": true,
    "lastCycle": "2026-08-13T08:00:14Z",
    "blocksCompacted": 12482,
    "blocksDeleted": 3820
  }
}

blocksDeleted > 0 is the proof retention is firing. A zero with lastCycle recent means either no blocks were old enough, or the compactor IAM role is missing s3:DeleteObject.

How it can fail

1. Compactor not deployed. In microservices mode, a missing compactor is silent: the ingester still writes blocks; the querier still serves queries; nothing is ever deleted. Symptom: bucket size grows unbounded.

2. Block_retention set to 0s. The default value means retain forever. A config that thinks it set 7 days but resolved to 0s produces the same outcome. Symptom: zero deletions in tempo_compactor_blocks_deleted_total.

3. Compactor IAM role missing s3:DeleteObject. The compactor lists blocks and decides they are eligible for deletion; the actual DeleteObjects call returns AccessDenied. Symptom: compactor log shows AccessDenied; bucket size flat-lines but does not shrink.

4. Compaction cycle slower than block growth. A small cluster with a 30-second compactor cycle and a 50k spans/sec ingest cannot compact fast enough; blocks accumulate faster than the compactor can delete them. Symptom: tempo_compactor_pending_tasks grows.

5. Per-tenant override typo. Tenant acme-prod gets retention 720h; a typo in the override file gives acmeprod retention 720h; the real tenant falls back to the default. Symptom: investigation at 60 days fails for what was meant to be a 30-day tenant.

6. Lifecycle policy on the bucket deletes before the compactor. An S3 lifecycle rule that expires objects after 30 days fights the compactor’s own deletion. The race is usually harmless, but can produce “block not found” errors during compaction. Symptom: compactor log shows intermittent NoSuchKey.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Is the compactor running? curl /ready on the compactor pod. If 404, the compactor is not deployed.
2. What does the config say? curl /config returns the active YAML. Confirm block_retention is the value you expect, in the right unit.
3. Did the last cycle delete anything? Look at tempo_compactor_blocks_deleted_total. If zero, the compactor has not yet had a block old enough.
4. Does the IAM role include delete? The same role that writes blocks must be able to delete them. Check the attached policy.

Security implications

• Retention as policy. Per-tenant overrides are a compliance instrument: regulated tenants can be forced to a 7-year retention, ephemeral tenants to 24 hours. Audit the overrides file the same way you audit IAM policies.
• Compactor credentials. The compactor needs read, write, list, and delete on the bucket. A bug that escalates this role to bucket-admin is a privilege escalation; scope the policy to the prefix Tempo owns.
• Hard delete vs soft delete. Tempo hard-deletes blocks. There is no “undelete” path through Tempo. A backup of the bucket (cross-region replication, snapshot) is the only recovery mechanism.

Performance implications

• Compactor CPU. The compactor holds blocks in memory while merging; large blocks (max_block_bytes: 100 GB) require matching heap. Default to 4 GB heap and raise with block-1.
• Bucket listing. Each compactor cycle lists every block under the prefix. With millions of blocks, the listing is the bottleneck. The fix is compaction: fewer, larger blocks list faster.
• Deletion throttling. S3 limits DeleteObjects to 1000 keys per call; GCS has similar batch limits. Tempo handles this with pagination but a very large batch can take time.

Production guidance

• Set block_retention to a value the team has reasoned about. 30 days is a reasonable production default; 90 days for regulated workloads.
• Run the compactor as its own StatefulSet, not as a sidecar. A stuck compactor should not take down ingest.
• Apply the S3 lifecycle rule to expire prefixes after the compactor’s retention has had time to delete. The race is usually harmless but adds noise.
• Alert on tempo_compactor_blocks_deleted_total == 0 over 24 hours. A non-zero value is the only proof retention is working.

Verification

You should now be able to answer:

• What does compactor.compaction.block_retention: 0s mean?
• Where is the per-tenant override file referenced from?
• Which IAM permission does the compactor need that the ingester does not?
• What is the difference between compaction and deletion in the compactor lifecycle?
• How do you prove retention is firing without waiting 90 days?

Quiz