Tempo Backup · Observability

The trace volume was high. The team had enabled Tempo’s block compression and shipped the bucket to S3 with versioning. The DR runbook said “restore from S3”. On the day of the test, the search index was missing. The trace blocks were there; the search data was not. The recovered Tempo could serve traces by trace ID; the search by service name returned nothing. The on-call engineer spent the next four hours discovering that the search data was in a separate bucket that the team had never enabled versioning on.

This lesson is the Tempo backup. The right shape is the same as Loki’s: object-store versioning plus cross-region replication, applied to every bucket the trace pipeline writes to. The trace blocks are one bucket; the search data is another; the metrics-generator state is a third if the metrics-generator is enabled. A “Tempo backup” that covers one bucket is a partial backup.

What it is

A Tempo backup is a copy of the durable state that Tempo needs to serve a trace query from a recovered cluster. The storage layers in modern Tempo:

Trace blocks. Compressed trace blocks in the object store. The primary source of truth for trace content. Default format is v2 (parquet) in current Tempo; older v1 is still common.
Search data. Per-tenant inverted index in the object store. Required for service-name and span-attribute search. Lives in a separate bucket.
Ingester WAL. Local-disk write-ahead log for in-flight trace blocks. Not the source of truth; optional.
Metrics-generator state. When the metrics-generator is enabled, it stores its derived metrics in a Prometheus TSDB on local disk, on the metrics-generator host.

The configuration (Tempo YAML, receivers, overrides) lives in Git. The “backup” of configuration is the Git history.

The discipline is to enable versioning and cross-region replication on every Tempo bucket, to persist the metrics-generator state to durable storage when the metrics-generator is enabled, and to prove the recovery path with a quarterly drill.

Why a sysadmin cares

Tempo is the most expensive observability component by bytes per second of input. Losing the trace store means:

Investigation evidence for the affected window is gone. Traces are the only signal that ties a request to its dependency graph.
Search by service name is lost if the search bucket is missing; the recovered Tempo requires knowing the trace ID to retrieve a trace.
The cost of the lost data is invisible until a slow query or a user-reported bug asks for it; the recovery window is silent.

A working backup turns a bucket loss into a recovery event. A partial backup turns the same loss into a partial investigation.

How it works

   Receivers (OTLP, Jaeger, Zipkin)
        |
        v
   Distributor
        |
        v
   Ingester (in-memory + optional WAL on local disk)
        |
        |  flush at block size or age threshold
        v
   Trace blocks (object store, primary)
        s3://tempo-traces-<account>-<region>/
            <tenant>/<block-id>.parquet
        |
        v
   Compactor (merges blocks, marks for deletion)
        |
        v
   Search data (object store, secondary)
        s3://tempo-search-<account>-<region>/
            <tenant>/<search-store>
        |
        v
   Querier (reads trace blocks + search index)
        |
        v
   Metrics-generator (optional)
        local-disk TSDB
        /var/tempo/metrics-generator/wal

The backup shape is the bucket-level shape:

   Primary buckets (3+)
   tempo-traces, tempo-search, tempo-blocks (if separate)
        |
        |  versioning: ON
        |  replication: cross-region, cross-account
        v
   DR buckets (one per primary, in separate account)
        |
        v
   Quarterly drill: query a known trace ID from the DR bucket
   against a temporary Tempo instance. Verify search by
   service name; verify trace-by-id retrieval.

The metrics-generator state, when enabled, needs a separate shape. Its TSDB can be snapshotted like a Prometheus TSDB and shipped to its own versioned bucket.

What versioning protects and what it does not in a Tempo bucket

Object-store versioning protects against:

Accidental overwrite. A PUT to an existing trace-block key keeps the previous version.
Accidental delete. A DELETE on a versioned key creates a delete marker; the previous version is recoverable.

Versioning does not protect against:

A missing search bucket. If the search bucket has versioning off (the default for new buckets), an operator overwrite loses the search index. The trace blocks are fine; the search functionality is gone.
Region-wide deletion by lifecycle. A misconfigured lifecycle rule on the search bucket expires non-current versions aggressively; the search index is permanently gone.
The ingester WAL as a substitute. The WAL is a recovery buffer for in-flight blocks. If the WAL host dies before the flush, the unflushed traces are gone. The WAL is not the backup.
Compactor-marked-but-not-yet-deleted blocks. The compactor marks blocks for deletion in a separate file; the actual delete happens later. A pull of the bucket before the delete sees the marked blocks; a pull after sees the deletion. The recovery path must run against the versioned bucket, not the live bucket.

The defence is to enable versioning on every Tempo bucket at provisioning time, before the first trace lands. A bucket that has ever been written to without versioning cannot retroactively version its existing keys.

How to configure it

The Tempo side — declare the storage paths:

# /etc/tempo/tempo.yaml
# SEVERITY: CONFIGURATION (reload)
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-primary-${AWS_REGION}
      endpoint: s3.${AWS_REGION}.amazonaws.com
      region: ${AWS_REGION}
      # SSE is configured at the bucket level. The KMS key is set
      # via the bucket policy, not the Tempo config.
  search:
    # The search store is separate from the trace store.
    # The bucket must have versioning on; see below.
    backend: s3
    s3:
      bucket: tempo-search-primary-${AWS_REGION}
      endpoint: s3.${AWS_REGION}.amazonaws.com
      region: ${AWS_REGION}

# WAL is on local disk. It is not a backup. It is a recovery
# buffer for in-flight traces.
wal:
  path: /var/tempo/wal

# Metrics-generator is optional. When enabled, the state is in
# a local-disk TSDB; snapshot it like a Prometheus TSDB.
metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/metrics-generator

The bucket side — versioning and replication on every bucket:

# CloudFormation / Terraform excerpt
# SEVERITY: CONFIGURATION
Resources:
  TempoTracesBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'tempo-traces-primary-${AWS::Region}'
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          # Trace blocks are large. The non-current expiration is
          # the recovery window. Hot tier: 30 days.
          - Id: TracesExpire
            NoncurrentVersionExpiration:
              NoncurrentDays: 30
            Status: Enabled
          # Move non-current to STANDARD_IA at 30 days; further
          # transition is rare for trace data because the
          # retention window is typically short.
          - Id: TracesIA
            Transitions:
              - StorageClass: STANDARD_IA
                TransitionInDays: 30
            Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
      ReplicationConfiguration:
        Role: !GetAtt BackupReplicationRole.Arn
        Rules:
          - Id: TracesToDR
            Status: Enabled
            Prefix: ''
            Destination:
              Bucket: !Sub 'arn:aws:s3:::tempo-traces-dr-${DRRegion}'

  TempoSearchBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'tempo-search-primary-${AWS::Region}'
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          # Search data is smaller per block but more index-y.
          # Keep non-current for 60 days; search is what an
          # investigator uses first.
          - Id: SearchExpire
            NoncurrentVersionExpiration:
              NoncurrentDays: 60
            Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
      ReplicationConfiguration:
        Role: !GetAtt BackupReplicationRole.Arn
        Rules:
          - Id: SearchToDR
            Status: Enabled
            Prefix: ''
            Destination:
              Bucket: !Sub 'arn:aws:s3:::tempo-search-dr-${DRRegion}'

The metrics-generator snapshot — when the metrics-generator is enabled:

#!/usr/bin/env bash
# SEVERITY: SERVICE-IMPACT (calls the metrics-generator admin API).
set -euo pipefail
MG_URL="http://tempo-metrics-generator.internal:3100"
SNAP_ROOT="/var/tempo/metrics-generator/snapshots"
BUCKET="s3://tempo-metricsgen-primary-${AWS_REGION}"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"

# 1. Take the snapshot through the metrics-generator admin API.
# The TSDB snapshot is consistent at the API level; copying the
# live directory would race.
SNAP_NAME=$(curl -fsS -X POST \
  "${MG_URL}/api/v1/admin/tsdb/snapshot" \
  | sed -E 's/.*"name":"([^"]+)".*/\1/')

# 2. Tar and ship.
tar -C "${SNAP_ROOT}" -czf "/tmp/${SNAP_NAME}.tar.gz" "${SNAP_NAME}"
AWS_PROFILE=tempo-backup aws s3 cp \
  --storage-class STANDARD_IA \
  --sse aws:kms \
  --sse-kms-key-id "${TEMPO_KMS_KEY}" \
  "/tmp/${SNAP_NAME}.tar.gz" \
  "${BUCKET}/${STAMP}/metricsgen.tar.gz"

rm -f "/tmp/${SNAP_NAME}.tar.gz"
curl -fsS -X DELETE \
  "${MG_URL}/api/v1/admin/tsdb/snapshot?name=${SNAP_NAME}" \
  >/dev/null

How to validate it

Top-level: every Tempo bucket is versioned and replicated.

# SEVERITY: READ-ONLY
for b in tempo-traces-primary-${AWS_REGION} \
         tempo-traces-dr-${DR_REGION} \
         tempo-search-primary-${AWS_REGION} \
         tempo-search-dr-${DR_REGION}; do
  AWS_PROFILE=tempo-backup aws s3api get-bucket-versioning \
    --bucket "$b" | jq '.Status // "Disabled"'
done
# Expected (illustrative): "Enabled" on each line.

Mid-level: a recent metrics-generator snapshot exists.

# SEVERITY: READ-ONLY
AWS_PROFILE=tempo-backup aws s3 ls \
  s3://tempo-metricsgen-primary-${AWS_REGION}/ \
  --recursive | sort | tail -1
# Expected:
# 2026-08-14 12:00:02    1048576 2026-08-14T120002Z/metricsgen.tar.gz

End-level: the drill served a known trace ID from the recovered Tempo.

# SEVERITY: READ-ONLY
cat /var/backups/tempo/drill/last-drill.txt
# Last successful restore drill of tempo: 2026-05-14,
# served trace-id abc123... and search by service name from the DR buckets.

The drill runbook in skeleton form:

# SEVERITY: SERVICE-IMPACT (boots a staging Tempo against the DR buckets)
STAGE=/opt/tempo-drill
mkdir -p "${STAGE}/config"
cat > "${STAGE}/config/tempo.yaml" <<EOF
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-dr-${DR_REGION}
      region: ${DR_REGION}
  search:
    backend: s3
    s3:
      bucket: tempo-search-dr-${DR_REGION}
      region: ${DR_REGION}
server:
  http_listen_port: 3201
EOF
docker run -d --name tempo-drill \
  -p 3201:3200 \
  -v "${STAGE}/config:/etc/tempo" \
  grafana/tempo:latest \
  -config.file=/etc/tempo/tempo.yaml
sleep 60
# Smoke test 1: search by service name.
curl -sG http://localhost:3201/api/search \
  --data-urlencode 'query={ resource.service.name = "tempo-drill" }' \
  --data-urlencode 'limit=1' \
  | jq '.traces | length'
# Expected: a positive integer; 0 means search is broken.
# Smoke test 2: trace by ID.
TRACE_ID=$(curl -sG http://localhost:3201/api/search \
  --data-urlencode 'query={ resource.service.name = "tempo-drill" }' \
  --data-urlencode 'limit=1' \
  | jq -r '.traces[0].traceID')
curl -s "http://localhost:3201/api/traces/${TRACE_ID}" \
  | jq '.spans | length'
# Expected: a positive integer; 0 means the trace block is missing.

How it can fail

Five failure modes recur in Tempo backup:

The search bucket is provisioned without versioning. The trace blocks bucket has versioning; the search bucket does not. An operator overwrite of the search index loses the service-name search. Symptom: TempoDrillRestoredSearch alert fires; the drill returns zero traces on service-name search.
The ingester WAL is treated as the backup. A runbook says “the WAL is the backup of the trace data”. The WAL host dies; the unflushed traces are gone. Symptom: TempoIngestedTraces and TempoQueriedTraces diverge by the unflushed amount.
The metrics-generator state is on local disk without snapshot. When the metrics-generator is enabled, the derived-metric TSDB lives in /var/tempo/metrics-generator. The host dies; the derived metrics are gone. Symptom: the tempo_* derived series disappear from Prometheus on restart.
The drill was never run. A quarterly restore drill on the calendar was scheduled for two years and never executed. Symptom: RestoreDrillOverdue alert fires for 700 days; the team treats it as background noise.
The schema version changed but the drill host is pinned. A Tempo upgrade introduced a new block format; the staging drill host still runs the old version. Symptom: the recovered Tempo logs unsupported block version and rejects the trace blocks.

How to troubleshoot it

The order is: are all buckets versioned, is the metrics-generator state durable, does the recovery path serve a known trace.

Are all buckets versioned? aws s3api get-bucket-versioning on every Tempo bucket. Missing versioning on the search bucket is the most common shape.
Is the metrics-generator state durable? Check the metrics-generator config and the snapshot job output. Local disk without snapshots is the failure shape.
Does the recovery path serve a known trace? Run the drill. If the drill has not run in the policy window, schedule it before any other change.
Are the metrics-generator admin API and the WAL on the same host? If the metrics-generator TSDB is on a different host from the WAL, the snapshot job must run against the metrics-generator host, not the Tempo ingester host.

What happens when the trace blocks bucket has versioning but the search bucket does not?

The drill boots a recovered Tempo against the DR buckets. The trace-by-id query works; the search-by-service-name query returns empty. The recovered Tempo cannot answer the question “show me all traces for service X in the last hour” — the question an investigator asks first. The on-call engineer can find traces by ID only; without the search index, they need to know the trace ID in advance, which they do not.

The fix is structural: provisioning must version every bucket in the trace pipeline. A “Tempo backup” policy that covers only the trace blocks bucket is incomplete. The alert on the search bucket versioning status catches the misconfiguration; the alert on the drill outcome catches the recovery failure.

Security implications

The trace blocks bucket holds every span, every tag, every attribute for the retention window. Traces often contain more sensitive data than logs (request bodies, query parameters, internal user IDs). Treat the bucket as production PII.
KMS encryption with a customer-managed key is mandatory. The key is rotated independently of the bucket.
The bucket is private. Public access is blocked at the bucket level and the account level. The replication role is the only role that crosses bucket boundaries.
The metrics-generator state can include derived business metrics. Treat the snapshot bucket as configuration data.
The drill staging host holds a copy of the trace store. Treat the staging host as production data; wipe on completion.

Performance implications

S3 versioning on the trace blocks bucket doubles storage cost for the same logical volume. Trace volume is high; the doubling is the largest line item in the observability bill. Use a short non-current expiration (7-14 days) for hot trace data and accept the smaller recovery window.
The search bucket is smaller per block but more numerous. The storage cost is meaningful; the lifecycle rules should transition non-current to STANDARD_IA early.
The metrics-generator snapshot is a few MB per snapshot. The cost is negligible.
A drill that downloads the DR buckets to a staging host transfers the full retention volume. The staging host’s network is sized for the largest realistic trace volume; a small staging host is the wrong choice for a drill.

Production guidance

Object-store versioning on every Tempo bucket (traces, search, metrics-generator if separate). Cross-region replication to a separate account.
KMS encryption with a customer-managed key. The key is rotated on a separate cadence from the bucket.
Lifecycle: 30 days non-current for trace blocks (hot tier), 60 days for search.
Metrics-generator state snapshot daily when enabled; ship to a versioned bucket.
Alert on every bucket’s versioning status. The search bucket is the easy one to forget.
Restore drill quarterly. Smoke test both search-by-service-name and trace-by-id. Document the result.
The backup role cannot delete. Lifecycle handles expiry.
The ingester WAL is not a backup. It is a recovery buffer for in-flight traces.

Verification

You should now be able to answer:

Which Tempo storage layers are durable (need backup) and which are derived or ephemeral?
Why is the search bucket the easy one to forget in a Tempo backup policy?
What is the role of the metrics-generator state snapshot, and when is it required?
Why is the ingester WAL not a backup?
What is the dual smoke test for a Tempo restore drill?