Runbook: Recover Tempo

“Recover Tempo” is not one procedure. Tempo has fewer moving parts than a log store - there is no index to rebuild, and blocks are immutable once written - but that simplicity comes with an unforgiving property: the object store is the only long-term copy of anything. Everything in this runbook is either about getting back to that store, or about not destroying it while trying.

   distributor        <- validates, rate limits, no durable state
        |
        v
   ingester ---> write-ahead log on local disk   <- the unflushed
        |                                            window, minutes
        v
   trace blocks ---> object store   <- canonical. Everything older
        |
        +--------> search store     <- a SEPARATE bucket. TraceQL
        |                              finds nothing without it
        v
   compactor       <- the ONLY component that deletes anything

Two lines in that diagram do most of the damage in practice. The compactor has delete authority over the canonical store. And the search store is a second bucket that a “restore the Tempo bucket” instruction silently omits, which is discovered later, by an investigator who has a service name and no trace ID.

Classify first: which layer was lost

Shape	What is happening	What it looks like	First move
WRITE PATH STALLED	Spans accepted, flush to the bucket failing	Ingester disk climbing; no new blocks appearing; queries fine	Read the storage error in the ingester log
INGESTER	The unflushed window is gone, or is replaying	Recent traces missing; historical fine; ingester not ready	Let replay finish; read the log first
OBJECT STORE	The bucket is gone, unreachable, or repointed	Historical queries empty; the last few minutes still answer	Stop the compactor, then verify the bucket
CONFIG	The binary is running, and running the wrong thing	Nothing arriving, nothing lost; everything reports ready	Validate the config; check what is bound

The first row is the one with a deadline. A stalled flush does not stop ingest immediately - the ingester keeps accepting spans and keeps writing to the write-ahead log - so the platform looks healthy while the disk fills. When the disk fills, the loss starts, and it starts at the newest data. Everything else on this list has already happened; that one is still happening.

The last row is the one people misdiagnose most confidently. Tempo ignores configuration keys it does not recognise rather than rejecting them, so a receiver block with a misspelled key produces a process that starts, registers its role, answers ready, and binds no port. Nothing was lost. Nothing is arriving. Every health check is green.

When this runbook applies, and when it does not

It applies when Tempo cannot return traces that were ingested, or cannot accept traces that should be ingestible, because a durability layer was lost or damaged.

It does not apply when:

The traces never arrived. If the distributor never received the spans, there is nothing to recover here. That is the missing-traces path, and it starts in the application process.
The window is outside retention. The compactor deleted it by policy. That is a retention decision to revisit, not an incident.
The query is wrong. An empty TraceQL result against a healthy cluster is a query problem, and running a recovery procedure against it makes things worse for no reason.
Tempo is merely slow. A querier scanning a wide time range across many small blocks is a compaction and capacity problem. Restarting components under that load turns slow into unavailable.

Blast radius

Action	Reversible?	What it costs if wrong
Reading readiness, status, counters and logs	n/a	Nothing
Stopping the compactor	Yes - start it again	Retention pauses; block count grows. Cheap
Expanding the ingester write-ahead log volume	Yes	Storage cost, and time you did not have
Restarting an ingester	Partly	Replay time, plus anything not yet in the write-ahead log
Moving a write-ahead log segment aside	No	Every trace in that segment, permanently
Repointing Tempo at a restored bucket	Depends	A stale snapshot silently sets your recovery point
Restoring in place over newer objects	No	The newer objects
Running the compactor mid-restore	No	Retention applied to gaps. Permanent deletion
Shortening retention to relieve pressure	No, once compaction runs	Data deleted by policy, on purpose, by you

Four rows are one-way doors. All four appear in the escalation path, because none of them should be walked through by one tired operator acting alone.

Step 1 - Classify, without changing anything

Read-only / Safethe reads that name the loss shape

# Substitute your own values before running:
TEMPO=http://tempo.observability.svc:3200
NS=observability

curl -s "$TEMPO/ready"

for role in distributor ingester querier compactor; do
printf '%s: ' "$role"
curl -s -o /dev/null -w '%{http_code}\n' "$TEMPO/$role/ready"
done

kubectl logs -n "$NS" -l component=ingester --tail=200 \
| grep -iE 'flush|wal|replay|storage|denied|error' | tail -20

ready
distributor: 200
ingester: 503
querier: 200
compactor: 200

Illustrative output

Readiness is role-aware and that is the point. The ingester is not ready until its write-ahead log replay finishes; the compactor is not ready until it has its lock. A check that treats any HTTP 200 from the front door as healthy misses both, and a 503 from the ingester during replay is correct behaviour rather than a fault.

Read-only / Safeis the write path stalled, and is anything arriving

TEMPO=http://tempo.observability.svc:3200

curl -s "$TEMPO/metrics" | grep -E '^tempo_ingester_(failed_flushes_total|local_blocks)'

curl -s "$TEMPO/metrics" | grep -E '^tempo_distributor_(spans_received_total|dropped_spans_total)'

tempo_ingester_failed_flushes_total 4821
tempo_ingester_local_blocks 39
tempo_distributor_spans_received_total 128420551
tempo_distributor_dropped_spans_total 0

Illustrative output

That combination is the signature of the first row of the table. Received climbing means spans are still arriving. Failed flushes climbing means none of them are reaching the bucket. Local blocks accumulating means the ingester is holding what it cannot flush. The diagnosis is not “Tempo is broken”; it is “the object store is refusing writes”, and the log carries the reason verbatim.

Step 2 - Stop the compactor, before anything touches a bucket

Service impact possiblethe cheapest irreversibility control in this runbook

# Substitute your own values before running:
NS=observability

kubectl scale -n "$NS" deployment/tempo-compactor --replicas=0

kubectl get pods -n "$NS" -l component=compactor

The cost of stopping the compactor is that retention pauses and the block count grows. Small blocks accumulating make queries slower. That is the whole bill, and it is trivially reversible.

Step 3 - WRITE PATH STALLED: read the storage error, then buy time

The stall is almost never inside Tempo. The three storage errors that produce an identical rising flush counter have three different owners:

Log line	Cause	Owner
`AccessDenied` on a put	The service account lost write permission, usually in a policy refactor	Whoever owns the bucket policy
A redirect on every request	The configured region or endpoint does not match the bucket	Whoever provisioned the bucket
A dial or DNS failure	Network policy, endpoint change, or a private-link path that moved	Networking

Read-only / Safeprove the bucket with Tempo permissions, not with yours

# Substitute your own values before running. Run this from a Tempo host
# or an exec into a Tempo pod, so it uses the same credentials Tempo has.
BUCKET=tempo-traces-prod
PREFIX=blocks/single-tenant/

aws s3 ls "s3://$BUCKET/$PREFIX" --recursive | tail -5

df -h /var/tempo/wal

Listing succeeding while a put fails is the shape to look for. It is the signature of an IAM policy written against the read path only, and it passes every check an operator runs today while breaking the flush that happens next.

Step 4 - INGESTER: let replay finish, and be explicit about the window

A restarting ingester replays its write-ahead log before serving. Replay time scales with the unflushed window, which is governed by max_block_duration and the flush interval. A long flush interval means a long write-ahead log, which means a long recovery - which is where your recovery time objective is actually decided, at configuration review, not during the incident.

Two distinct quantities are at stake and they are worth separating out loud:

In memory, not yet in the write-ahead log. Gone on any process loss. Measured in seconds.
In the write-ahead log, not yet flushed. Returns if the volume survived; gone with the volume if it did not. Measured in whatever max_block_duration is set to.

If the ingester refuses to start and logs a corruption error against a segment, you are at a one-way door. Move the segment aside rather than deleting it: the ingester starts with a truncated log, every trace in that segment is lost, and the file is the only artefact that can later answer what was in the gap.

Step 5 - OBJECT STORE: verify before you repoint, and restore both stores

A restored bucket is not a recovered bucket. Verify that objects covering the affected window are actually present, under the prefix and in the region Tempo is configured to use.

Read-only / Safedoes the restored bucket contain the window you need

# Substitute your own values before running:
BUCKET=tempo-traces-dr
PREFIX=blocks/single-tenant/

aws s3 ls "s3://$BUCKET/$PREFIX" --recursive --summarize | tail -3

Restore into a separate prefix or bucket and verify there, whenever the object store offers that option. Restoring in place over newer objects is not reversible, and the newer objects are the ones from the window you care about most.

The trace-block bucket is half the store

Trace blocks give back retrieval by trace ID. The search store is what lets TraceQL find a trace when nobody has the ID - which is the normal case, because the whole reason to open Tempo is usually that you have a service name and a time range.

A recovery that restores only the block bucket passes every check an operator naturally runs: the pods are ready, a known trace ID comes back, the counters are climbing. It fails at the first real investigation, hours or days later, and by then the recovery is considered finished and nobody is looking.

Enumerate every bucket the pipeline writes to before restoring anything, and verify each one with a query shape that actually reads it. If the metrics generator is enabled, its state is a third store, on local disk, and it is not in either bucket.

Step 6 - CONFIG: validate before you start the binary

Read-only / SafeTempo ignores keys it does not recognise

tempo-cli validate-config /etc/tempo/tempo.yaml

# What is actually bound, which is a different question:
ss -lntp | grep -E '3200|4317|4318'

config is valid: /etc/tempo/tempo.yaml

Illustrative output

The validator checks that the YAML is structurally valid. It does not check that the bucket is reachable, that the credentials work, or that a port is free. Those are three separate questions and the second command answers only the last of them.

Deploy the known-good commit from version control rather than hand-editing the running configuration, then diff the two. A hand edit made during an incident is the drift that causes the next one.

Step 7 - Bring it back in order, then prove it twice

Object store reachable, then ingesters, then queriers and the query frontend, then the compactor last.

Read-only / Safetwo different proofs, and you need both

# Substitute your own values before running:
TEMPO=http://tempo.observability.svc:3200
OLD_TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736

# 1. The write path and the read path together, on new data.
otel-cli span export --endpoint tempo.observability.svc:4317 \
--service recovery-check --name "post-recovery synthetic"

# 2. The historical read path, which is the only one that touches
#    the object store. Use an ID recorded BEFORE the incident.
curl -s "$TEMPO/api/traces/$OLD_TRACE_ID" \
| jq '[.resourceSpans[].scopeSpans[].spans[]] | length'

# 3. Search, which reads the search store rather than the blocks.
curl -sG "$TEMPO/api/search" \
--data-urlencode 'q={ resource.service.name = "checkout" }' \
--data-urlencode 'limit=10' | jq '.traces | length'

These three prove different things and none substitutes for another. The synthetic trace proves receiver, distributor, ingester and querier. The historical lookup proves the block bucket. The search proves the search store. Stopping after the first is the most common way a “finished” Tempo recovery is discovered to be half-finished.

Step 8 - Write down the gap, then restart the compactor

The gap boundary is the deliverable. Start, end, tenants, and whether the traces were lost or merely delayed - those are four separate facts and the fourth is the one people assume rather than establish.

Then restart the compactor and watch its first cycle complete. That cycle is the first time retention is applied to whatever shape the bucket is now in, and it is the last chance to notice that the shape is wrong while the mistake is still recoverable.

Holding is a first-class option

Not every Tempo recovery should be finished at 03:00. If the block bucket is intact and only the search store is missing, the platform can serve trace-by-ID lookups immediately while the search restore runs in daylight. If the write path is stalled and the volume has been expanded, ingest is safe and the bucket permissions can wait for the team that owns them.

Holding needs the same two things every deferral needs: a named owner and an end time. It also needs the gap published, because a partially recovered Tempo that nobody has described is worse than an obviously broken one - people trust it.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

Classify first: which layer was lost

When this runbook applies, and when it does not

Blast radius

Step 1 - Classify, without changing anything

Step 2 - Stop the compactor, before anything touches a bucket

Step 3 - WRITE PATH STALLED: read the storage error, then buy time

Step 4 - INGESTER: let replay finish, and be explicit about the window

Step 5 - OBJECT STORE: verify before you repoint, and restore both stores

Step 6 - CONFIG: validate before you start the binary

Step 7 - Bring it back in order, then prove it twice

Step 8 - Write down the gap, then restart the compactor

Holding is a first-class option

References