Missing Traces Anatomy · Observability

A user reports a failed payment at 14:21. The application log contains a trace_id and a span_id. At 14:23 the on-call engineer opens Tempo and searches for that trace ID. Tempo returns trace not found. The trace ID is correct. The application wrote the trace ID into the log because the SDK generated one. So a span was emitted. So a trace existed. And yet Tempo says no. Five links must succeed for a span to land in Tempo and become a queryable trace; one of them is broken. This lesson is the chain, and the order to check it.

What it is

A “missing trace” is any trace that the application emitted but the operator cannot retrieve from Tempo by its trace ID. The five links in the chain are:

  +-------+    +-----------+    +-----------+    +---------+    +-------+
  |  SDK  | -> | propagates| -> | collector | -> | exporter| -> | Tempo |
  +-------+    +-----------+    +-----------+    +---------+    +-------+
       A            B               C               D              E

Link A — SDK running. The application’s process has loaded and initialised an OpenTelemetry SDK and the SDK has at least one Tracer registered.
Link B — Propagation intact. The traceparent header defined by W3C Trace Context is being injected on outbound calls and extracted on inbound ones across every service boundary in the path.
Link C — Collector reachable. The SDK or an in-process agent (Grafana Alloy sidecar, otel-collector-contrib agent) is accepting OTLP over a TCP or Unix socket.
Link D — Exporter healthy. The collector’s exporter pipeline is sending the batch to Tempo over OTLP gRPC and the batch is being acknowledged.
Link E — Sampling kept it. The sampler in the SDK or in a tail-sampling processor made the decision to keep the trace, and the trace ID was recorded in the kept set.

A trace can be missing at Tempo because any one of A through E failed. Each link has a distinct metric, log line, or command that confirms it. The cost of a wrong diagnosis is an hour of restarting things that were not the cause.

Why a sysadmin cares

Production investigations time out. A paged engineer who opens a trace to find the slow span, and gets trace not found, has spent twenty seconds on the investigation and ninety seconds on the wrong path. The question “is the trace in Tempo or not?” must answer in under sixty seconds, not under ten minutes. The diagnostic order is what makes that answer cheap.

The same five-link chain breaks in slightly different ways for each language SDK, each collector flavour (core, contrib, Grafana Alloy), and each sampling topology (head, tail, remote). The order of the diagnostic is the same. The evidence at each link is what changes.

How it works

Each link has a small surface. The diagnostic at each link is one command and one decision:

  Link    Cheap check                                       If the check fails...
  ----    ----------------------------------------------    ----------------------------
   A      Search the service logs for the SDK banner;       The SDK is not running.
          grep the process for "OpenTelemetry SDK".
   B      tcpdump the boundary between service A and B      The traceparent header
          and look for the traceparent header.              is missing.
   C      curl -v otel-collector:4317/ from the SDK host.   The collector is
                                                           unreachable.
   D      collector exporter view in the logs; the          The exporter cannot
          otelcol_exporter_sent_spans counter.                deliver.
   E      Check the SDK sampler probability; check the      The sampler dropped the
          tail_sampling_decision counter.                    trace.

The check at link A is the cheapest; the check at link E is the most expensive. The diagnostic order is the cost order.

Link A in detail

The SDK is running when an SDK Tracer has been created and at least one span has been started. The SDK prints a startup banner at INFO level for most languages. Examples:

  # Java agent
  INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller -
  OpenTelemetry Java agent v2.10.0 loaded

  # Node SDK
  {"name":"@opentelemetry/instrumentation-http","msg":"patching"}

  # Python opentelemetry-instrument
  Opentelemetry instrumentation: enabled for http, flask, sqlalchemy

  # Go otel/otel SDK
  (no banner; SDK is silent. Confirm by reading the TracerProvider
   is set on the global at process start.)

If the banner is absent or the SDK has not been initialised in main(), the application runs without instrumentation. No spans are emitted. Every later link is irrelevant until A is fixed.

Link B in detail

A traceparent header on every outbound HTTP or gRPC call is the visible signal of intact propagation. The header value follows the W3C Trace Context format:

  traceparent: 00-<32 hex trace-id>-<16 hex span-id>-<2 hex flags>
  tracestate:  vendor1=value1,vendor2=value2

A boundary that strips the header (a proxy, a service mesh sidecar with Propagation disabled, a load balancer with a header allow-list) breaks the chain at link B. The trace is emitted on both sides, but the two halves do not share a trace ID. Tempo shows two separate traces, neither of which the operator can correlate to the user’s request.

Link C in detail

The collector is reachable when the OTLP exporter can open a gRPC stream to it. The check is a TCP probe:

  nc -zv otel-collector.observability.svc 4317

A connection refused is link C. A connection accepted but the exporter retries is link C with a transport-level failure (mismatch on TLS, mTLS, framing). The collector itself exposes otelcol_receiver_accepted_spans to confirm it is processing received bytes.

Link D in detail

The exporter is healthy when its outgoing OTLP stream is acknowledged by Tempo. The check is the collector’s otelcol_exporter_sent_spans counter, paired with otelcol_exporter_send_failed_spans. A flat sent_spans counter with a rising send_failed_spans counter is link D.

Link E in detail

The sampler keeps the trace when its decision is RecordAndSample or equivalent. A head sampler in the SDK runs at span start and decides per trace. A tail sampler in the collector runs after span collection and decides per trace. The check is the collector tail-sampling decision counter or the SDK’s sampler log line.

How to configure it

The “configuration” of a missing trace is the configuration that proves each link. The five checks have to be in place before any incident: SDK self-observability, propagation verification, collector probe, exporter counter, sampling log. The five checks are not the trace pipeline; they are the meta-pipeline that catches a broken trace pipeline.

SDK self-observability

# Java agent: -Dotel.metrics.exporter=prometheus \
#             -Dotel.exporter.prometheus.port=9464
# Python:      OTEL_METRICS_EXPORTER=prometheus \
#              OTEL_EXPORTER_PROMETHEUS_PORT=9464
# Node:        /metrics endpoint enabled by the SDK
# Go:          otelhttp.WithMeterProvider and prometheus exporter

A typical SDK self-observability scrape config in Grafana Alloy:

  prometheus.scrape "otel_sdk" {
    targets = [{
      __address__ = "checkout-svc:9464",
      job         = "otel-sdk",
      service     = "checkout-svc",
    }]
    forward_to      = [prometheus.remote_write.default.receiver]
    scrape_interval = "30s"
    metrics_path    = "/metrics"
  }

Severity: CONFIGURATION. Restart the application after enabling the SDK’s Prometheus exporter.

Collector probe

A Kubernetes probe against the collector’s OTLP gRPC port:

  livenessProbe:
    tcpSocket:
      port: 4317
    initialDelaySeconds: 10
    periodSeconds: 30
  readinessProbe:
    grpc:
      port: 4317
    periodSeconds: 5

Severity: CONFIGURATION. Reapply the manifest; the kubelet takes the change on next period.

Sampling decision log

A collector with tail sampling logs every decision at DEBUG:

  processors:
    tail_sampling:
      decision_wait: 10s
      num_traces: 50000
      expected_new_traces_per_sec: 200
      policies:
        - name: errors
          type: status_code
          status_code: { status_codes: [ERROR] }
        - name: slow
          type: latency
          latency: { threshold_ms: 1500 }

Severity: CONFIGURATION. Restart the collector to apply.

How to validate it

Severity: READ-ONLY.

The five-link check at the moment a trace is missing:

  # Link A — SDK running
  kubectl logs deploy/checkout-svc --since=10m | \
    grep -iE 'opentelemetry|otel' | head -5
  # Opentelemetry SDK 1.42.0 initialised
  # TracerProvider[ io.opentelemetry.sdk.trace.SdkTracerProvider@... ]

  # Link A (alternative) — process envvar present
  kubectl exec deploy/checkout-svc -- \
    printenv | grep -E '^OTEL_'
  # OTEL_SERVICE_NAME=checkout-svc
  # OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
  # OTEL_EXPORTER_OTLP_PROTOCOL=grpc

  # Link B — traceparent on outbound calls
  kubectl exec deploy/checkout-svc -- \
    curl -sI http://payment-svc:8080/healthz | grep -i traceparent
  # traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

  # Link C — collector reachable
  kubectl exec deploy/checkout-svc -- \
    nc -zv otel-collector.observability.svc 4317
  # otel-collector.observability.svc (172.20.4.18:4317) open

  # Link D — collector exporter OK
  curl -s http://otel-collector.observability.svc:8889/metrics \
    | grep -E '^otelcol_exporter_(sent|send_failed)_spans'
  # otelcol_exporter_sent_spans{exporter="otlp/tempo"} 128421
  # otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} 0

  # Link E — sampling kept it
  curl -s http://otel-collector.observability.svc:8889/metrics \
    | grep -E 'tail_sampling_(decision|sampled|dropped)'
  # otelcol_processor_tail_sampling_decision_timer_count{policy="errors",decision="sampled"} 4211
  # otelcol_processor_tail_sampling_decision_timer_count{policy="slow",decision="sampled"} 982
  # otelcol_processor_tail_sampling_decision_timer_count{policy="errors",decision="dropped"} 113
  # otelcol_processor_tail_sampling_decision_timer_count{policy="slow",decision="dropped"} 41028

A dropped counter that dominates the sampled counter is link E.

How it can fail

The five failure shapes, one per link. The diagnostic at each link is the cheapest signal that names the shape.

Link A — SDK not initialised. The application was deployed without the agent, or the agent was disabled by a feature flag, or the SDK’s bootstrap code was removed in a refactor. Symptom: no span output anywhere; the otel_sdk scrape returns only zero counters; no trace_id ever appears in the application logs. This is the single most common cause.
Link A — SDK initialised but no exporter wired. The SDK prints its banner, registers a Tracer, but no BatchSpanProcessor is attached. Symptom: SDK counters (otel.sdk.span.started) rise; no spans reach the collector; the exporter log shows “no exporter configured”.
Link B — Propagation stripped at a mesh sidecar. The service mesh sidecar has propagation.allowlist set to only allow specific headers; traceparent is not in the list. Symptom: traces are present on each side of the boundary; the trace IDs do not match; Tempo shows two separate traces for the same request.
Link C — Collector is down. The collector pod restarted and is stuck in CrashLoopBackOff. Symptom: nc -zv otel-collector 4317 returns Connection refused; otelcol_receiver_accepted_spans is flat.
Link D — Exporter’s TLS handshake fails. The collector’s otlp/tempo exporter is configured with TLS, but Tempo is using a certificate signed by a CA the collector does not trust. Symptom: collector log shows x509: certificate signed by unknown authority; otelcol_exporter_send_failed_spans rises.
Link E — Tail sampler drops everything. The tail sampler’s decision_wait is shorter than the slowest span in the trace. Spans arrive after the decision; the sampler marks the trace dropped because it has no spans to evaluate. Symptom: tail_sampling_decision_timer_count rises uniformly; Tempo receives only the traces that finished before the wait.

How to troubleshoot it

The diagnostic order is link A first. The temptation to start at link D is wrong; the exporter is downstream of the SDK, and a missing SDK makes the exporter vacuously healthy.

Link A. Does the application process have an SDK banner or environment variables? If not, the SDK is not running; fix the SDK bootstrap.
Link A, alt. Does the SDK’s self-observability scrape show non-zero counters? If the counters are flat at zero but the banner is present, the SDK is initialised but no spans are being created. Inspect the code that starts a span.
Link B. Hit a known endpoint with curl -v and grep for traceparent. If the header is missing on outbound, the propagation is broken at this service; if it is present outbound but absent inbound on the next service, the next service’s propagation is broken.
Link C. Is the collector reachable? Is the OTLP endpoint DNS-resolvable from the application host? Is port 4317 open in the network policy?
Link D. Are spans arriving at the collector (otelcol_receiver_accepted_spans)? Are they being sent to Tempo (otelcol_exporter_sent_spans)? Are they being rejected (otelcol_exporter_send_failed_spans)?
Link E. Is the sampler keeping the trace? Read the sampling policy log; check the decision counter.

Security implications

A “missing trace” is rarely a security incident on its own, but the five-link chain intersects with security boundaries:

Link C is a network boundary. The collector’s OTLP gRPC port is exposed to every instrumented host. mTLS or network policy is required to keep untrusted hosts from flooding the receiver.
Link D carries span data over the network to Tempo. Spans contain attribute values; an attribute that holds a customer email or a session token is a confidentiality leak. Filter at the collector’s attributes processor before exporting.
Link E (tail sampling) keeps traces by policy. A policy that drops traces containing PII is a privacy control; a policy that keeps every trace is a privacy leak.

Performance implications

The five-link diagnostic adds no load to the production pipeline. The SelfObservability metrics are scraped at 30-second intervals and the collector’s /metrics endpoint is read on demand. The cost is the time the on-call engineer spends reading the metrics; the right order turns that time into under a minute.

The bigger performance trap is configuring a full OTLP pipeline and then failing to flush. A BatchSpanProcessor buffers spans on a schedule and a size; a process that exits without flushing drops every buffered span. The fix is the SDK’s shutdown hook, attached to a SIGTERM handler.

Production guidance

Always enable the SDK’s self-observability exporter on a known port and scrape it. The metric set is the cheapest link-A check that exists.
Always run tcpdump or curl -v against the boundary between two instrumented services during a smoke test. The traceparent header is the link-B check.
Always check otelcol_exporter_send_failed_spans before every other collector metric. The exporter’s failure counter is the link-D check.
Always set a non-zero decision_wait on the tail sampler and never shorter than the slowest expected trace.
Always document the chain. The diagram in this lesson is the one to print.

Verification

You should now be able to answer:

Name the five links in the chain that turns a span into a queryable trace.
Which link is the most common cause of a missing trace?
Which metric distinguishes “SDK not initialised” from “SDK initialised but no exporter wired”?
What header is the visible signal of intact propagation?
Why does the diagnostic order start at link A?