Runbook: Investigate Tempo Ingestion Failure

Somebody has opened a trace view and found nothing recent. That single symptom is produced by four separate failures with four different owners, four different urgencies, and one of them on a timer.

The reflex is to restart Tempo. It is the wrong first move in three of the four cases and actively destructive in the fourth.

Four failures, one symptom - and the discriminator is free

Spans travel receiver, then distributor, then ingester, then object store. The pipeline can break at each hop, and from Grafana all four look identical:

A. Nothing is arriving. The receiver is not bound, the client cannot reach it, or it is speaking the wrong protocol.
B. Spans arrive and are refused. A limit, an oversized batch, or a validation rejection. Clients see 429 or ResourceExhausted.
C. Spans are accepted but cannot reach the ingesters. The ring is short of members and the write cannot reach quorum.
D. Spans are accepted, held, and never flushed. The object store is refusing writes. Recent traces are queryable, older ones are not, and the ingester WAL is growing.

Reading two counters twice, thirty seconds apart, tells you which one you have. That costs nothing, changes nothing, and is the difference between a targeted fix and an hour of guessing.

D is the only one that gets worse while you think about it.

When to use this runbook

Trace search returns nothing recent, across services rather than for one trace.
Collectors or Alloy report export failures to Tempo.
The ingest-rate panel has been flat since a deploy, a certificate change, or a cloud policy change.
An ingester is reporting disk pressure on its WAL filesystem.

It does not apply when:

One trace ID will not resolve. That is the five-link chain from SDK to Tempo - instrumentation, propagation, collector, exporter, sampling
- and it belongs to the missing-traces runbook. Most of those links are upstream of anything Tempo can see.
Traces are arriving but queries are slow or empty. The write path is working; the read path is the problem, and restarting ingesters will not help it.
The bucket itself has been lost. That is Tempo recovery, not ingestion. Ingestion can be perfectly healthy while every historical block is gone.
The applications are down. They are not emitting. Tempo is behaving correctly and this is the wrong page.

Blast radius

Tempo ingestion failing is not, on its own, a customer-facing outage. No request fails because a span was dropped. What is lost is investigation capability, and it is lost silently: the traces for the window are simply not there when somebody comes looking, usually during a different incident.

Two ways it escalates into something larger:

The WAL disk. If spans are being accepted and cannot be flushed, the ingester keeps writing to its write-ahead log. That grows until the filesystem fills, at which point the ingester stops accepting spans - and a full filesystem is rarely a problem confined to one process.
Back-pressure into the senders. A distributor returning 429 makes collectors retry and buffer. Their queues grow, their memory grows, and on a shared agent that memory is being taken from whatever else the agent is doing.

The remediation has a blast radius of its own. Restarting an ingester that is holding unflushed spans risks those spans. Raising an ingestion limit moves the pressure to the ingesters and the bucket, which is what the limit was protecting.

Step 1: The reading that classifies the failure

Read-only / Safetwo readings, thirty seconds apart

TEMPO=http://tempo.internal:3200

# List the families rather than one name. Metric names have changed
# across Tempo releases; read what THIS build exposes.
curl -s "$TEMPO/metrics" | grep -E '^tempo_(distributor|ingester)_' | sort > /tmp/t1
sleep 30
curl -s "$TEMPO/metrics" | grep -E '^tempo_(distributor|ingester)_' | sort > /tmp/t2

diff /tmp/t1 /tmp/t2 | grep -E 'spans_received|bytes_received|traces_created|append_failures|failed_flushes'

The counters that matter are the distributor’s received counter, the ingester-side received or traces-created counter, the distributor’s append-failure counter, and the ingester’s failed-flush counter. Read them as a set:

Distributor received	Ingester side	Append failures	Failed flushes	Branch
flat	flat	flat	flat	A - nothing arriving
rising	flat or lagging	flat	flat	B - arriving and refused
rising	flat	rising	flat	C - ring cannot take the write
rising	rising	flat	rising	D - not flushing to the bucket

Two readings, not one. A counter with a large value proves the pipeline worked at some point in the past, which is exactly the fact that misleads people at 03:00.

Branch A: Nothing is arriving

Read-only / Safeis the front door open, and to which protocol

# Is anything bound? OTLP gRPC 4317, OTLP HTTP 4318,
# Jaeger gRPC 14250, Zipkin 9411.
ss -tlnp | grep -E '4317|4318|14250|9411'

# Did the receiver stanza parse at all? A misspelled key is skipped
# silently - there is no error, the receiver simply does not exist.
journalctl -u tempo --since '1 hour ago' | grep -iE 'receiver|listen|error'

# Tempo's own readiness.
curl -s http://tempo.internal:3200/ready

Three causes, in the order they are worth checking:

The receiver is not configured. A tempo.yaml whose distributor.receivers block was never enabled, or was enabled on the wrong role in a microservices deployment, gives clients a connection refused. In microservices mode the receivers belong to the distributor; a config copied onto a querier binds nothing.

The protocol does not match. An OTLP exporter aimed at 4317 when only the HTTP receiver is enabled establishes a TCP connection and then fails the gRPC handshake. From the server the traffic is invisible; from the client the error is explicit. Read the client’s log - it is the better evidence here.

The network path is broken. A policy change, a certificate, a NAT. The receiver is bound and healthy and nothing reaches it. Test from a client host rather than from the Tempo host, because testing from the Tempo host proves only that Tempo can talk to itself.

Branch B: Arriving and refused

Read the client error before touching the server, because it names the cause precisely:

429 - a per-tenant ingestion limit. The four caps are ingestion_rate_limit_bytes, ingestion_burst_size_bytes, max_traces_per_user and max_bytes_per_trace.
ResourceExhausted - the OTLP batch is larger than max_recv_msg_size_mib on the receiver. One noisy service drops everything while the rest of the fleet is fine.
A validation rejection - a trace without a usable ID, a timestamp far outside the acceptable range, or an attribute value large enough to be a bug rather than data.

The instinct at this point is to raise the limit. Resist it for one minute and ask what changed: a new SDK release that double-emits, a sampling policy that was turned off, a retry loop. The limit did its job

it kept one tenant from taking the ingesters down for everyone. Raising it moves the failure to the ingesters and the bucket, where it is more expensive and harder to reverse.

Where the cause is genuinely more traffic than the platform was sized for, the cheaper direction is usually sampling at the source. That is the tenant’s change, with the tenant’s knowledge.

Branch C: The ring cannot take the write

Read-only / Safeingester ring membership

curl -s http://tempo.internal:3200/ingester/ring | jq '.'
curl -s http://tempo.internal:3200/ingester/ready

{"name":"ingester","members":[
{"addr":"tempo-0:3200","state":"ACTIVE"},
{"addr":"tempo-1:3200","state":"ACTIVE"},
{"addr":"tempo-2:3200","state":"ACTIVE"}]}

Illustrative output

Every ingester should be present and ACTIVE. With replication_factor: 3, the distributor forwards each trace to three ingesters and needs a quorum to acknowledge; a ring two members short cannot satisfy that, and the distributor fails the append no matter how healthy the surviving ingesters look individually.

A ring that is churning rather than short usually means the lifecycler or the memberlist KV store, and a rolling restart in progress is the commonest benign explanation. Confirm nobody is mid-deploy before treating it as a fault.

Branch D: Accepted, held, never flushed

This is the branch that hides. Spans are accepted, so the counters look right. Recent traces are queryable, because they are still in the head block. Only traces older than the flush interval are missing, and nobody notices until they go looking for something from this morning.

Meanwhile the WAL grows.

Read-only / Safeis the bucket actually receiving blocks

# The newest object under the tenant prefix. A timestamp older than
# the flush interval means writes are not landing.
aws s3 ls s3://tempo-traces-prod/blocks/single-tenant/1/ --recursive \
| sort | tail -3

# Tempo's own view: failed flushes, and blocks flushed.
curl -s http://tempo.internal:3200/metrics \
| grep -E 'tempo_ingester_(failed_flushes|blocks_flushed)_total'

# What the process says when the write fails.
journalctl -u tempo --since '1 hour ago' \
| grep -iE 'AccessDenied|SlowDown|flush|s3|bucket'

The usual cause is not inside Tempo at all. A credential that lost s3:PutObject during a policy refactor produces exactly this shape: the ingester retries with backoff, reports success on the accept path, and nothing lands. AccessDenied in the log is the confirmation.

The other shapes worth recognising: SlowDown (or ServerBusy, or a 429 from the object store) means the bucket is throttling and the fix is rate rather than permissions; a connection refused means endpoint or DNS; and a lifecycle policy that has archived recent blocks produces read failures rather than write failures, which is a different problem wearing similar clothes.

The restart decision

The distributor is stateless. It holds nothing, and restarting it costs only the in-flight batches, which the senders retry. If a distributor change is needed, make it.

The ingester is not. It holds spans that exist nowhere else until they are flushed. Stop it gracefully so it can flush and so the WAL is consistent for replay; never kill it; and never restart it while the object store is the broken component, because the flush it needs to perform is the thing that cannot happen.

If the pressure is genuinely intolerable and the bucket is still broken, that is an escalation, not a restart.

Verification

Read-only / Safea synthetic span, end to end

# Emit through the same path a real client uses - not straight at the
# ingester, which would prove nothing about the receiver.
otel-cli span --service runbook-check --name ingest-probe \
--endpoint tempo.internal:4317

# Then fetch it back. Substitute the trace ID the command printed:
TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo.internal:3200/api/traces/$TRACE_ID" | jq '.batches | length'

# And confirm it is searchable, which exercises the query path too.
curl -sG http://tempo.internal:3200/api/search \
--data-urlencode 'q={ resource.service.name = "runbook-check" }' \
--data-urlencode 'limit=5' | jq '.traces | length'

Then re-read the counters. A moving metric plus a retrievable trace plus a draining WAL is the complete claim; any one of them on its own is not.

Finally, look at the senders. Tempo accepting spans while the collectors are still failing to export means the fix landed somewhere other than the fault.

Common patterns

Symptom	Likely branch	Resolution
Connection refused from every client	A	Receiver not bound, or configured on the wrong role
Client reports a gRPC handshake failure	A	OTLP aimed at a protocol that is not enabled
Client reports `429`	B	Per-tenant ingestion limit; check the tenant before the limit
Client reports `ResourceExhausted`	B	Batch larger than `max_recv_msg_size_mib`
Everything counted against tenant `anonymous`	B	`X-Scope-OrgID` missing or wrong case; overrides never match
Distributor receiving, append failures rising	C	Ring short of members; check for a stuck rolling restart
Recent traces fine, older ones gone, WAL growing	D	Flush path broken; check the bucket credential before Tempo
Bucket returning `SlowDown`	D	Throttling, not permissions; reduce rate or spread the prefix
Ingestion fine after a restart, back within an hour	D, unresolved	The restart cleared the symptom; the flush path is still broken

Escalation

Escalate when:

The bucket is refusing writes. Every minute of that is WAL growth.
The receiver is healthy and clients cannot reach it - that is the network team’s evidence to read, not yours.
One tenant is over its limits: their sampling, their decision.
A global limit would have to change. That is capacity planning, not triage.
The WAL filesystem is above 80 percent and climbing.
Ingest is spiking from a source nobody recognises. The receiver ports accept unauthenticated traffic by default, and that makes an unexplained spike a security question.

Why this matters

The expensive part of this incident is not the fix. Each of the four branches has a small, obvious remedy once you know which one you are in: bind a receiver, correct a header, restore a permission, wait out a rolling restart. The expensive part is picking the branch, and the reflex

restart it and see - picks at random.

That reflex is specifically dangerous here because the four branches are not equally forgiving. Three of them are stateless: nothing is at risk while you investigate, and a restart is merely useless. The fourth is holding data that exists nowhere else, on a disk that is filling, and a restart into a broken flush path is the one action that turns lost time into lost spans.

Two counter readings thirty seconds apart cost nothing and separate them. It is a small discipline with an asymmetric payoff, and it is the same discipline the rest of this runbook is built on: read the running process before changing it, and let the evidence choose the branch rather than the other way round.

The other habit worth taking away is smaller and comes up constantly. When a metric name returns nothing, that is not the same as the metric being zero - and in a system where the names have shifted across releases, treating those two as the same fact is how a careful operator reaches a confident wrong conclusion.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation