Tempo Storage Scaling

A team kept 30 days of traces. The monthly storage bill was 6x what they had budgeted. The investigation found that the compactor was running, the bucket was private, the IAM role was scoped. The miss was that no lifecycle policy was configured. The compactor marked blocks for deletion past block_retention: 48h; the bucket retained them indefinitely because nothing actually deleted them. The fix was a one-line lifecycle policy.

This lesson is the discipline of sizing the storage layer for Tempo: bucket growth, the compaction cycle, tiering, and the cost of trace retention.

What it is

Tempo stores blocks in object storage. The growth pattern has three layers:

• Writes. Every flush produces one or more blocks in the bucket.
• Compaction. The compactor merges small blocks into larger ones, reducing block count.
• Retention. The compactor marks blocks past block_retention for deletion. The bucket lifecycle policy reaps the bytes.

Storage cost has two components:

• Bytes. The on-disk size of the bucket. The dominant driver of the storage bill.
• Block count. The number of objects in the bucket. The dominant driver of query latency.

The two are decoupled. A bucket with 1 TiB of data in 100 blocks is cheaper to query than a bucket with 100 GiB in 10 million blocks. A configuration that optimises bytes (high compression, short retention) can still produce a query-latency problem if the compactor is not running.

Why a sysadmin cares

Three operational pains are specific to storage scaling:

1. Storage bill grows faster than expected. A retention window that is too long, a compactor that is not running, or a missing lifecycle policy produces a bucket that retains data indefinitely. The bill reflects weeks of data, not days.
2. Query latency rises with bucket size. Even with compaction, the querier must list blocks per query. A bucket with millions of objects spends most of its query time in ListObjectsV2 calls.
3. Tiering mismatch. A bucket that stores 30-day-old traces on Standard class storage is paying for fast I/O on data nobody queries. A lifecycle policy that moves old blocks to Glacier or Infrequent Access cuts the bill dramatically.

How it works

The storage pipeline has three stages and three operators:

  Ingester flush
       |
       v
  +---------------------------+
  | Bucket (S3 / GCS / Azure) |   <-- ingester writes
  +---------------------------+
       |
       |  compactor lists
       v
  +---------------------------+
  | Compactor                 |   <-- marks for deletion
  +---------------------------+
       |
       |  bucket lifecycle policy
       v
  +---------------------------+
  | Bytes reaped              |   <-- actually deleted
  +---------------------------+

The three operators are independent. The compactor and the ingester know nothing about the lifecycle policy; the lifecycle policy knows nothing about Tempo. The retention window must be consistent across all three.

The compactor marks blocks for deletion by writing a marker file in the tenant’s markers/ prefix. The lifecycle policy is configured to expire objects older than block_retention (the same value as the compactor uses). When the lifecycle policy fires, it deletes the bytes; the compactor then removes the marker on its next cycle.

How to configure it

A production storage config pins the bucket, the compactor retention, and the lifecycle policy. The Tempo YAML:

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
    wal:
      path: /var/tempo/wal

compactor:
  compaction:
    block_retention: 48h
    compaction_window: 1h

The matching S3 lifecycle policy (Terraform or aws-cli):

cat > /tmp/lifecycle.json <<'EOF'
{
  "Rules": [
    {
      "ID": "tempo-traces-tiered-retention",
      "Status": "Enabled",
      "Filter": { "Prefix": "blocks/" },
      "Transitions": [
        {
          "Days": 1,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 7,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": { "Days": 30 }
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket tempo-traces-prod \
  --lifecycle-configuration file:///tmp/lifecycle.json

Three details to call out:

• block_retention and the lifecycle Expiration.Days must agree. A compactor block_retention: 48h paired with a lifecycle Expiration.Days: 30 produces a bucket that retains blocks for 30 days; the compactor marks them at 2 days; the lifecycle waits 28 more days to actually reap them. This is the silent cost bug described in the opening.
• Tiering saves money. A bucket that stores 30-day-old traces on Standard class storage is paying for fast I/O on data nobody queries. The transition to Standard-IA after 1 day and to Glacier after 7 days cuts the bill by 60-80% without changing query coverage for recent data.
• Lifecycle is independent of the compactor. A lifecycle policy that fires before block_retention deletes blocks before the compactor has merged them. The result is small blocks that survived the lifecycle; the compactor then has nothing to merge.

Severity: CONFIGURATION. Lifecycle policies take up to 48 hours to take effect after creation. Plan ahead.

How to validate it

Severity: READ-ONLY.

1. Confirm the bucket is reachable from the ingester and the compactor:

aws s3 ls s3://tempo-traces-prod/blocks/ --recursive | head -5
# 2026-08-12 10:42:01      12345 blocks/ingester/tenant/01H.../meta.json

2. Confirm the lifecycle policy is active:

aws s3api get-bucket-lifecycle-configuration \
  --bucket tempo-traces-prod | jq .
# {
#   "Rules": [
#     {
#       "ID": "tempo-traces-tiered-retention",
#       "Status": "Enabled",
#       "Filter": { "Prefix": "blocks/" },
#       "Transitions": [...],
#       "Expiration": { "Days": 30 }
#     }
#   ]
# }

3. Confirm the bucket size and block count:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix 'blocks/' \
  --max-items 0 \
  --query 'Length' \
  | jq .
# 482301

aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 \
  --metric-name BucketSizeBytes \
  --dimensions Name=BucketName,Value=tempo-traces-prod \
               Name=StorageType,Value=StandardStorage \
  --start-time -P7D \
  --period 86400 \
  --statistics Average | jq .

4. Confirm the compactor retention and the lifecycle agree:

# compactor side
yq '.compactor.compaction.block_retention' /etc/tempo/tempo.yaml
# 48h

# lifecycle side
aws s3api get-bucket-lifecycle-configuration \
  --bucket tempo-traces-prod \
  | jq '.Rules[0].Expiration.Days'
# 30

A compactor block_retention: 48h and a lifecycle Expiration.Days: 30 is a mismatch; the lifecycle waits longer than the compactor. The bucket retains blocks for 30 days. The two values must be derived from the same source.

How it can fail

Six shapes appear repeatedly:

1. Lifecycle policy missing. The compactor marks blocks for deletion; nothing reaps the bytes. Symptom is the bucket growing monotonically past block_retention and the storage bill rising.
2. Lifecycle expiration too short. The lifecycle fires before the compactor merges blocks. Symptom is small blocks surviving past the compaction window and the compactor having nothing to merge.
3. Lifecycle on the metrics bucket. The metrics-generator writes service-graph metrics to a separate bucket. A lifecycle policy that targets the trace bucket by prefix but is applied to both buckets deletes metrics early. Symptom is missing service-graph data after the lifecycle expires.
4. Tiering to Glacier too aggressive. A bucket that transitions to Glacier after 1 day forces every query against blocks older than 1 day to wait for Glacier retrieval (minutes). Symptom is query p99 latency of minutes for traces older than a day.
5. Bucket in a region that costs more than expected. A bucket created in us-east-1 while the rest of the platform runs in eu-west-1 produces cross-region data transfer on every block fetch. Symptom is the data-transfer line item on the bill.
6. Compactor retention and lifecycle out of sync. A block_retention: 48h and an Expiration.Days: 7 produces a bucket that retains blocks for 7 days; the compactor stops trying to mark them after 2 days. Symptom is the compactor metric flat at zero but the bucket growing.

How to troubleshoot it

The diagnostic order:

1. Is the compactor running? /ready and the compactor metric.
2. Is the lifecycle policy active? aws s3api get-bucket-lifecycle-configuration.
3. Do the retention values agree? The compactor block_retention and the lifecycle Expiration.Days must be derived from the same source.
4. How many objects in the bucket? aws s3api list-objects-v2 with --max-items 0.
5. What is the bucket size? CloudWatch BucketSizeBytes.
6. Are blocks the right tier? S3 inventory or storage-class metrics in CloudWatch.

Security implications

Storage has three attack surfaces:

• Bucket credentials. A leaked AWS key with s3:GetObject on the Tempo bucket is a trace-data breach. Use scoped credentials and rotate them.
• Bucket policy. A bucket policy that allows public access exposes every trace to the internet. Default to private.
• Lifecycle audit. A lifecycle policy that deletes blocks before block_retention expires early destroys data. Treat the policy as a write-once artefact; review changes.

Performance implications

Storage performance has two axes:

• Query latency. Driven by block count, not byte size. Keep block count bounded by running the compactor.
• Storage cost. Driven by byte size, not block count. Reduce bytes by tiering to cheaper classes and by short retention.

The two axes decouple. A platform that optimises cost without optimising block count pays the difference in query latency. A platform that optimises query latency without optimising cost pays the difference in the bill. Both must be addressed.

Production guidance

• Configure the lifecycle policy before the first production workload. A bucket that grows without a lifecycle policy is hard to clean up later.
• Tier aggressively. Recent blocks live on Standard; 1-7 day blocks on Standard-IA; older blocks on Glacier.
• Run the compactor. The bucket growth without compaction is the single most common storage cost bug.
• Monitor both bytes and block count. The bill is bytes; the query latency is block count.

Verification

You should now be able to answer:

• What are the three operators that must agree on the retention window?
• Why does block count matter as much as byte size?
• What is the cost of a missing lifecycle policy?
• How do you tier a Tempo bucket across hot, warm, and cold storage?
• Why does the compactor block_retention and the lifecycle Expiration.Days have to be derived from the same source?

Quiz