Tempo Storage

At 03:14 the on-call engineer notices that trace lookups return empty for incidents from the previous 36 hours. Logs show tempo_querier_search_results_total is non-zero but the API returns nothing. The block listing against the S3 bucket shows that the most recent block folder is dated two days ago. The ingester is healthy and flush_to_storage is firing. The problem is that the IAM role attached to the ingester pods lost the s3:PutObject permission during a policy refactor. The ingester reports success because its write retry buffer is full of pending objects; the writes are not actually landing.

Tempo stores every trace block in object storage. The choice of backend and the bucket layout are not abstract concerns; they are the only persistent copy of the data.

What it is

Tempo’s storage layer is a single backend (storage.trace) that points at an object store. Tempo writes trace blocks — immutable, content-addressed units of a few hundred megabytes containing compressed spans — into the bucket. The block filename embeds the trace ID prefix range and the tenant ID, which is what enables index-less lookup.

Supported backends:

• S3 (AWS, MinIO, Ceph Rados Gateway) — most common.
• GCS (Google Cloud Storage).
• Azure Blob Storage.
• Local filesystem (single-node lab use only).

Tempo does not write a query index (by design; the design is called “index-less”). It relies on the bucket layout for search and on the block contents for retrieval.

Why a sysadmin cares

• Cost. Trace data is the largest of the three telemetry signals at typical span sizes. At 50k spans/sec with 4 KB per span after compression, the daily write volume is around 17 TB. A small misconfiguration on retention or lifecycle policy is a five-figure monthly bill.
• Availability. Tempo has no local durability for the trace data; if the bucket is unavailable, queries fail even though the Tempo pods are healthy.
• Region / latency. Trace queries hit the bucket on every lookup. A bucket in eu-west-1 queried from us-east-1 adds 80-120 ms to every trace fetch.

How it works

Block lifecycle

The ingester buffers spans in memory, then flushes them to object storage on two triggers:

• Time. ingester.flush_to_storage (default 15 minutes) rolls the in-memory buffer into a block and uploads.
• Size. ingester.max_block_duration (default 30 minutes) bounds how long a single block spans.

   In-memory trace buffer
            |
            v  (every flush_to_storage)
   +------------------+
   | trace block      |  ---> s3://bucket/blocks/.../tenant/<id>/<block-id>/
   |   - header       |
   |   - index        |
   |   - data         |
   |   - bloom        |
   +------------------+
            |
            v  (compactor)
   compacted / merged blocks (deleted originals)

Bucket layout

Tempo writes under a configurable prefix. The default in recent releases is blocks/. Each block lives under the operator (typically single-tenant), tenant, block ID, and shard directories:

s3://tempo-traces/
  blocks/
    single-tenant/                       <-- operator
      1/                                 <-- tenant
        00000000-0000-...                <-- block ID
          1e0a6b2f-.../data.zng
          1e0a6b2f-.../index
          1e0a6b2f-.../meta.json
          1e0a6b2f-.../bloom...

The block ID is a ULID. The shard directories inside a block spread the load on the bucket and let the querier fetch only the shards that overlap the trace ID range being searched.

The prefix blocks/ is configurable:

storage:
  trace:
    backend: s3
    wal:
      path: /var/tempo/wal
    s3:
      bucket: tempo-traces
      prefix: blocks

Per-tenant structure

In multi-tenant mode, the tenant ID is the next level below the operator. A X-Scope-OrgID header on the OTLP request routes the span into that tenant’s folder. The querier, on lookup, scans only the tenant folder that the request’s auth context identifies.

Under the hood

How to configure it

S3 (AWS)

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
      session_token: ${AWS_SESSION_TOKEN}    # for STS / IRSA

      # Storage class hints. Defaults to STANDARD.
      storage_class: STANDARD_IA

      # Force the region — Tempo does not always trust the bucket.
      forcepathstyle: false

      # HTTP client tuning
      http:
        idle_conn_timeout: 90s
        response_header_timeout: 30s

Severity: CONFIGURATION. Restart the ingester, querier, and compactor to pick up new credentials.

For production on EKS, prefer IRSA over static keys: annotate the Tempo service account with the role ARN and remove access_key and secret_key. The session_token is populated by the AWS SDK.

GCS

storage:
  trace:
    backend: gcs
    gcs:
      bucket_name: tempo-traces-prod
      chunk_buffer_size: 10
      enable_managed_identity: true    # on GKE with Workload Identity

Azure Blob

storage:
  trace:
    backend: azure
    azure:
      container_name: tempo-traces
      storage_account_name: tempotracesprod
      storage_account_key: ${AZURE_STORAGE_KEY}
      use_managed_identity: true        # on AKS with Workload Identity
      endpoint_suffix: core.windows.net
      # force_path_style is irrelevant for Azure.

MinIO

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio.storage.svc:9000
      access_key: tempo
      secret_key: ${MINIO_PASSWORD}
      insecure: true                    # HTTP, not HTTPS
      forcepathstyle: true              # MinIO requires this

Severity: CONFIGURATION. MinIO’s path-style bucket access is required; forcepathstyle: true is mandatory or the endpoint is mis-parsed as a sub-domain.

How to validate it

Severity: READ-ONLY.

# 1. Confirm Tempo can list the bucket (uses Tempo's own perms)
curl -s http://tempo:3200/api/status | jq .

# 2. List blocks directly to confirm writes
aws s3 ls s3://tempo-traces-prod/blocks/single-tenant/1/ \
    --recursive --summarize | head -20

# 3. Confirm a specific block is reachable
aws s3 cp s3://tempo-traces-prod/blocks/.../meta.json - | jq .

Real output:

$ curl -s http://tempo:3200/api/status | jq .
{
  "version": "1.5.0",
  "storage": "s3",
  "ingester": { "healthy": true }
}

$ aws s3 ls s3://tempo-traces-prod/blocks/single-tenant/1/ \
    --recursive --summarize | tail -3
Object total: 47820
Total size: 5.4 TiB

A 404 on a block that Tempo expects means the bucket is reachable but the IAM role lacks s3:GetObject. A connection-refused means the endpoint or DNS is wrong.

How it can fail

1. IAM credential lost permission. The ingester retries the upload with exponential backoff; the in-memory buffer (max_traces_per_user) fills; new writes start failing with distributor_dropped_spans_total. Symptom: zero new blocks in the bucket; Tempo log shows AccessDenied.

2. Bucket in a different region than the Tempo pods. Each block fetch is a cross-region transfer; every query is slow; bandwidth costs balloon. Symptom: trace UI loads but each trace fetch takes seconds.

3. Lifecycle policy moves blocks to Glacier before block_retention expires. The compactor cannot delete the block (or must wait a restore cycle); queries fail with restore in progress errors. Symptom: random “trace not found” for blocks that exist but are archived.

4. forcepathstyle missing on MinIO. Tempo tries to use virtual-hosted style; MinIO returns 307 redirects that the AWS SDK mishandles. Symptom: writes succeed on a small block but fail on a large one with InvalidArgument.

5. Bucket prefix collision with Loki. A shared bucket with a Loki and Tempo prefix that both expect blocks/ is not actually possible (they use different filenames), but a shared bucket where one prefix contains the other leads to the querier listing objects that belong to the other service. Symptom: corrupt trace fetches that the application logs as invalid.

6. Azure managed identity not yet assigned at pod startup. The Tempo pod starts before the AKS Workload Identity webhook injects the token; the bucket listing fails with AuthenticationFailed. Symptom: intermittent cold-start failures; resolves after pod restarts.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Is the bucket reachable? aws s3 ls s3://bucket/ from the Tempo host (or pod). If this fails, Tempo will fail too.
2. Does the IAM role have read AND write? Check the role’s policy directly. s3:GetObject, s3:PutObject, and s3:ListBucket are the minimum for a working Tempo.
3. Are blocks being written? aws s3 ls --recursive s3://bucket/blocks/ | wc -l. A flat number across hours means writes are stuck.
4. Is the compactor running? A common silent failure is the compactor not having started because of a bad config block, leaving blocks accumulating without lifecycle application.

Security implications

• Credentials. The ingester, querier, and compactor all need bucket access. Use workload identity (IRSA on EKS, Workload Identity on GKE / AKS) rather than static keys.
• Bucket policy. Apply the principle of least privilege: the Tempo role gets only the prefixes it owns. The read-only Grafana data source can use a separate, restricted role.
• Encryption. Enable SSE-S3 / SSE-KMS on the bucket. Tempo does not encrypt spans client-side; the bucket policy is the line of defence.
• Block listing. A s3:ListBucket grant allows the credential holder to enumerate all tenants. In multi-tenant mode, this is information disclosure; restrict by IAM condition or per-tenant prefixes.

Performance implications

• Block size. Larger blocks reduce listing overhead but increase blast radius per block (a bad block affects more queries). The default flush_to_storage of 15 minutes is a reasonable compromise.
• Backend latency. S3 cross-region adds ~80-120 ms per block read. Keep the bucket in the same region as Tempo.
• Compactor pressure. A bucket with millions of small blocks is slow to list and slow to compact. The compactor’s job includes merging small blocks; ensure it runs.
• Cost. Approximate: at 4 KB compressed per span, 50k spans/sec, 7-day retention, the bucket is ~30 TB. On S3 Standard-IA at $0.0125/GiB/month, this is ~$375/month. Hot-tier or multi-region replicas multiply this.

Production guidance

• Run the bucket in the same region as the Tempo cluster.
• Use IRSA / Workload Identity; rotate keys out of YAML.
• Enable SSE-KMS with a customer-managed key.
• Apply an S3 lifecycle policy that moves blocks older than 7 days to Glacier IR, after Tempo’s compactor has had time to delete them. The compactor must finish before the lifecycle rule fires.
• Monitor tempo_ingester_blocks_flushed_total and tempo_compactor_blocks_compacted_total.

Verification

You should now be able to answer:

• What is the role of the bucket prefix in Tempo’s block layout?
• Which three IAM permissions are minimum for a working Tempo?
• Why does Tempo not maintain a separate index, and what replaces it?
• What does forcepathstyle: true do and which backend requires it?
• What is the per-tenant folder level directly below the operator?

Quiz