Runbook: Investigate Missing Traces

1 · Prerequisites

Confirm every item is in place before any state change.

• Missing Traces Anatomy
• Trace Propagation Test
• Tempo Architecture Overview
• Tempo Receivers
• TraceQL Introduction
• Tempo Querying
• Read access to the application logs, to the collector internal-telemetry endpoint, and to the Tempo query API
• The service name as the SDK reports it in the service.name resource attribute, which is frequently not the name humans use for the service
• A contact for whoever owns the ingress proxy or service mesh, because a stripped traceparent header is almost never fixable by the service team alone

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · The trace ID is in hand, copied rather than retyped, and is 32 hexadecimal characters. A truncated or transposed ID returns exactly the same not-found as a genuinely missing trace, and it is the single cheapest thing to rule out.
• · The scope is established before any theory: one trace missing, every trace from one service missing, or Tempo empty for every service. The three have different first moves and the first move for one wastes the window for the others.
• · It is established whether this ever worked. A service that has never produced a trace is a deployment problem, not an incident, and it does not belong in an incident channel at 03:00.
• · Tempo itself is confirmed to be answering: /ready on the HTTP port, plus a retrieval of a trace ID known to have been present before the suspect window. Without that second half, a broken query path is indistinguishable from a broken write path.
• · The age of the trace is compared against compactor.compaction.block_retention. A trace deleted by retention policy is not missing; it is expired, and no step in this runbook will bring it back.
• · The age of the trace is also compared against the export path latency. The SDK batch processor and the collector batch processor each add delay, so a trace queried seconds after the request has plausibly not been written yet. Wait before concluding.
• · The service name is confirmed as the SDK emits it, from OTEL_SERVICE_NAME or the resource attributes on a span that did arrive. Searching TraceQL for the name on the org chart returns nothing and proves nothing.
• · Every step in the diagnosis below is read-only. It is worth saying out loud before starting, because the reflex under pressure is to restart the collector, and that destroys the evidence at link D without touching the cause.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Establish the scope first. One trace, one service, or everything. Everything at once points at the collector, at Tempo, or at a shared credential rather than at any application, and starting at an application in that case burns the first twenty minutes.
2. 2Link A - confirm the SDK is running inside the application process. Look for the SDK startup banner in the service log and for the OTEL_ environment variables in the running process. No SDK means no span was ever created, and every later link is vacuously healthy. This is the most common answer and it is the cheapest to check.
3. 3Link A, second half - confirm the SDK is producing spans, not merely loaded. An SDK that initialises without a span processor attached prints its banner, registers a tracer, and exports nothing. The SDK self-observability scrape separates the two: counters flat at zero with a banner present means the exporter was never wired.
4. 4Link B - confirm the traceparent header survives every hop. Send one request through the entry point with a known traceparent and count the spans that come back under that trace ID. A count of one means the first hop after the entry point dropped the header. Check the count, not the presence of the trace.
5. 5Link B, second half - identify which hop dropped it. Compare the unique set of service.name values in the returned trace against the expected call path. The first service missing from the set is the boundary to investigate: an ingress proxy header allow-list, a mesh sidecar header policy, or an async queue that carries no context.
6. 6Link C - confirm the collector is reachable from the application host or pod. A TCP probe to the OTLP port from inside the client network namespace. Connection refused is the collector; a connection that opens and then fails the gRPC handshake is a protocol or TLS mismatch, which looks like the same symptom and has a different owner.
7. 7Link C, second half - confirm the collector is actually accepting the spans. otelcol_receiver_accepted_spans climbing proves that bytes are arriving and being decoded. Flat at zero with the port open means the client is connecting to a receiver that is not configured for the protocol it is speaking.
8. 8Link D - confirm the exporter is delivering to Tempo. Read otelcol_exporter_sent_spans against otelcol_exporter_send_failed_spans and otelcol_exporter_queue_size together. Sent flat with failed climbing is a delivery failure; sent flat with queue_size at its capacity is back-pressure, and the two are fixed in different places.
9. 9Link D, second half - read the collector log for the reason. The counters say that delivery failed; only the log says why. A certificate signed by unknown authority, a ResourceExhausted on message size, and a DNS failure all produce the same rising counter.
10. 10Link E - confirm the sampler kept this trace. Read the sampling decision counters and compare kept against dropped. A tail sampler whose decision_wait is shorter than the slowest span in the trace drops traces that were complete and correct, and it does so consistently rather than randomly, which is what makes it look like a storage problem.
11. 11Tempo side - confirm the distributor received and did not drop. tempo_distributor_spans_received_total minus tempo_distributor_dropped_spans_total is the delta that proves Tempo took the spans. A rising dropped counter is a Tempo-side limit or a rejected payload, and it moves the investigation to the ingestion runbook.
12. 12Name the link before fixing anything. Write down A, B, C, D or E and the evidence for it. Every fix below restarts a process or changes a shared configuration, and applying one to the wrong link produces a restart, no improvement, and a lost hour.
13. 13Apply the fix that matches the link, and re-run the propagation test to prove it. The synthetic request with a known trace ID is the only check that exercises all five links together. Readiness endpoints and healthy counters, in any combination, do not.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The specific trace ID that started this investigation returns spans from the Tempo trace-by-ID endpoint. Not "traces are appearing" - that one ID, retrieved and counted.
• ✓The span count in the returned trace matches the expected hop count for that request path. A trace that exists with one span is a link-B failure that a presence check reports as success.
• ✓The unique set of service.name resource attributes in the trace matches the expected call path end to end, including the last hop. Verifying at the entry point proves only that the entry point is instrumented.
• ✓otelcol_exporter_sent_spans is climbing while otelcol_exporter_send_failed_spans stays flat, and otelcol_exporter_queue_size is well below its configured capacity rather than pinned at it.
• ✓otelcol_receiver_accepted_spans is climbing for the receiver and transport the client actually uses, checked by label rather than in aggregate.
• ✓tempo_distributor_spans_received_total is climbing and tempo_distributor_dropped_spans_total is not. A non-zero delta between them is the only proof that Tempo accepted what the collector sent.
• ✓The scheduled propagation test passes, with the expected span count, on a run that started after the fix rather than before it.
• ✓The written diagnosis names one link and one boundary. "It was flaky" and "it came back on its own" are not diagnoses and guarantee the same hour is spent again.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Every diagnostic step is read-only and has nothing to undo. The items below are cleanup for fixes applied under pressure, which is where the real risk in this runbook lives.
• ↶If the sampling rate was raised to catch a trace, put it back. A head sampler left at 100 percent quietly multiplies ingest cost, storage cost and collector memory, and nobody notices until the next capacity review.
• ↶If a debug or logging exporter was added to the collector to see spans on the way through, remove it. It writes span attribute values into the collector log, which is the fastest way to move customer data into a log store that was never scoped to hold it.
• ↶If TLS verification was disabled on an exporter to get past a handshake error, re-enable it and fix the trust bundle instead. The disabled setting survives the incident, is invisible in every dashboard, and turns the span path into an unauthenticated one.
• ↶If a traceparent allow-list entry was added by hand at a proxy, get it into the configuration that builds that proxy. Header allow-lists are regenerated on the next deploy and the fix disappears without warning.
• ↶If an application was restarted by hand with the instrumentation agent attached, that restart is not the fix. The fix is the deployment manifest; until then the next rollout removes the instrumentation again.
• ↶If per-tenant or receiver limits were raised to absorb a burst, lower them once the backlog drains and record what the burst actually was. A limit raised during an incident and never reviewed is how the next capacity surprise is built.

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate to the team that owns the service when the diagnosis is link A. The SDK bootstrap, the agent attachment and the shutdown flush all live in their build and their deployment, and no platform-side change fixes any of them.
• · Escalate to the owner of the ingress proxy or the service mesh when the diagnosis is link B. A header allow-list is a platform control, and the service team cannot see it, let alone change it.
• · Escalate to networking when the collector port is unreachable from the client network namespace but reachable from elsewhere. That is a policy or routing question, and collector configuration changes will only obscure it.
• · Escalate to the platform team when tempo_distributor_dropped_spans_total is the rising counter. At that point the spans reached Tempo and Tempo refused them, which is a limits and capacity question rather than a pipeline one.
• · Escalate to security immediately if reading a trace exposes attribute values that should not be in telemetry - credentials, tokens, full request bodies or personal data. That is a disclosure with its own clock, independent of whether the trace was found.
• · Escalate when every service lost traces at the same moment and the collector checks out. Simultaneity is evidence: many services do not break together, but one shared credential, one certificate or one collector deployment does.
• · Escalate when the missing window covers an incident that is still open. Traces are the signal that ties a request to its dependency graph, and an incident commander needs to know explicitly that the graph is unavailable rather than empty.

References

1. Tempo: HTTP API, including trace-by-ID lookup
2. Tempo: TraceQL
3. W3C Trace Context
4. OpenTelemetry: context propagation
5. OpenTelemetry Collector: configuration
6. OpenTelemetry Collector: internal telemetry
7. OpenTelemetry Collector Contrib: tail sampling processor
8. OpenTelemetry: sampling