Missing Spans

The on-call engineer opens Tempo for a trace ID lifted from a failed checkout. The trace is there. It has three spans: API gateway, inventory, pricing. The inventory call had a child span for the SQL query. The pricing call has no children at all. The warehouse picklist service, which the inventory service is documented to call, does not appear anywhere in the trace. Either the call did not happen, or the spans did not arrive.

This is the lesson. A trace with fewer spans than the architecture diagram predicts is the most common trace quality problem in production. The diagnosis is a ladder, not a guess.

What it is

A missing span is a unit of work the on-call engineer expects to see in a trace and does not. The phrase covers two distinct failures:

• Not generated. The instrumented code path either did not run, or ran without an active span context. The SDK never created the span. Nothing left the process.
• Dropped in transit. The SDK created the span and emitted it on the wire, but the span did not reach Tempo. The collector may have rejected it, the exporter may have timed out, the receiver buffer may have overflowed.

The two failures look identical from the Tempo UI. The diagnostic ladder separates them.

Why a sysadmin cares

Three production payoffs ride on a complete trace.

1. Latency attribution. When one service is slow, the team needs to know whether the cost is the service, a downstream dependency, or a message-bus wait. Missing spans hide one of those three possibilities.
2. Failure boundary identification. When the inventory call returns 503, the team needs to know whether the inventory service itself failed or whether a downstream call from inventory (the warehouse picklist) timed out. The missing warehouse span leaves the question open.
3. Capacity planning. The histogram in Tempo is built from the span durations the team does have. Missing spans bias the picture. Latency looks lower than it is. Capacity looks looser than it is.

The cost of a missing span is not visible from the trace UI. The cost is paid in the next incident, when the on-call engineer cannot find the slow dependency.

How it works — the mental model

A trace is built by three components working in series: the SDK in the application, the collector pipeline in the middle, and the Tempo backend at the end.

Application
  +--- Span A (generated, emitted)
  +--- Span B (generated, emitted)
  +--- Span C (never generated - bug in code path)
  +--- Span D (generated, lost in collector batch - queue overflow)
  +--- Span E (generated, exporter timed out - network or backend down)
         |
         v
  OpenTelemetry Collector
    receivers -> processors -> exporters
         |
         v
  Tempo (ingester -> block -> querier)

The three failure bands each leave a different fingerprint.

• Not generated: no metric on the collector for that span. The application process did not invoke the SDK on that code path.
• Lost in collector: the collector self-metrics (otelcol_exporter_queue_size, otelcol_exporter_dropped_spans) increment. The spans arrived at the receiver but did not leave the exporter.
• Exporter failed: the exporter self-metric otelcol_exporter_send_failed_spans increments. The exporter attempted to send and the receiver (Tempo) rejected or timed out.

The diagnostic order walks those bands from inside the application outward.

How to configure it

The OpenTelemetry Collector is the place to make span loss observable. The minimum configuration for diagnostics:

# /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

exporters:
  otlp/tempo:
    endpoint: tempo.distribution.svc.cluster.local:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 30s
        max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888   # expose Prometheus metrics on :8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Three lines matter for the diagnosis:

• service.telemetry.metrics.address exposes the collector’s own metrics on a Prometheus scrape endpoint. Without this the team is guessing.
• exporters.otlp.sending_queue.retry_on_failure.enabled: true enables the retry. Without it, one Tempo blip drops the batch.
• processors.batch.send_batch_size: 8192 raises the default from 512. The trade-off is larger per-flush payloads against fewer flushes per second.

The Grafana Alloy equivalent, if the deployment uses Alloy as the agent on each host:

// /etc/alloy/config.river
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }

  output {
    metrics = [otelcol.exporter.prometheus.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  timeout = "5s"
  send_batch_size = 8192

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.distribution.svc.cluster.local:4317"
    tls { insecure = true }
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint { url = "http://mimir.distribution.svc.cluster.local:9009/api/v1/push" }
}

The pattern is the same. The names are different.

How to validate it

The diagnostic ladder runs from inside the application outward. Run each command in order. Stop when the answer explains the missing span.

# 1. Is the SDK initialised at all? Check the application log.
kubectl logs deploy/checkout -c app | grep -i "opentelemetry\|tracer\|sdk"
# Expected (healthy):
#   {"level":"info","msg":"tracer provider initialised",
#    "endpoint":"alloy:4317"}
# Unhealthy:
#   (nothing — the SDK never logged an init line, or it logged
#    a warning that the OTLPSpanExporter could not reach the endpoint)

# 2. Is the collector running and accepting spans?
curl -sf http://alloy:8888/metrics | grep otelcol_receiver_accepted_spans
# otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 18432
# (numbers climbing = receiver is healthy)

# 3. Are spans being forwarded to Tempo, or dropped in the batch?
curl -sf http://alloy:8888/metrics | grep -E "otelcol_exporter_(sent|dropped|failed)_spans"
# otelcol_exporter_sent_spans{exporter="otlp/tempo"} 18430
# otelcol_exporter_dropped_spans{exporter="otlp/tempo"} 0
# otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} 2
# (sent - failed - dropped should equal accepted, modulo the in-flight batch)

# 4. Does Tempo have the trace at all?
tctl trace search --service=checkout --limit=20 --since=1h | grep -c "$TRACE_ID"
# 1 (healthy) or 0 (Tempo never received it)

# 5. If Tempo has the trace, is it complete?
tctl trace show "$TRACE_ID" | jq '.spans | length'
# Compare to the expected count from the architecture diagram.

The fifth command is the answer. If the count matches the expectation, the diagnosis stops at the application (a code path that did not create a span). If the count is lower, the diagnosis continues into the collector and Tempo.

How it can fail

Six recurring failure modes.

1. SDK not initialised. The application imports opentelemetry-sdk but never calls TracerProviderBuilder().build() in main(). No tracer is registered. The auto-instrumentation libraries fall back to a no-op tracer. Symptom: zero spans from the service in Tempo; application logs contain no OpenTelemetry init line.
2. Exporter endpoint unreachable. The SDK is initialised with OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317, but the host cannot resolve alloy (DNS, NetworkPolicy, missing Service). The SDK retries forever, queueing spans until max_queue_size is full, then drops them. Symptom: the application log shows repeated connection-refused errors; Tempo shows nothing for that service.
3. Batch processor silently dropping under load. The queue overflows under burst load. The SDK logs the drop at ERROR level but the line is buried in noise. Symptom: Tempo shows a subset of traces during the burst window — typically only the longest ones, because shorter ones dropped first.
4. Receiver protocol mismatch. The application sends OTLP/gRPC but the collector is configured for OTLP/HTTP only (or vice versa). The TCP connection succeeds, the protocol handshake fails, the spans are silently discarded. Symptom: otelcol_receiver_refused_spans increments on the collector; the application sees no error.
5. Tempo ingester down or restarting. The collector accepts spans, queues them in the exporter, attempts to send, Tempo returns 503. With retry disabled the spans are dropped on the first failure. Symptom: otelcol_exporter_send_failed_spans increments sharply; Tempo’s own tempo_ingester_traces_created_total rate drops.
6. The SDK was initialised but the service is the wrong service. A staging build of the service is deployed to a production namespace. The OTel init line is absent because the staging image strips telemetry. Symptom: zero spans from one service name; all other services still emit. The Tempo search { resource.service.name = "expected-service" } returns nothing.

How to troubleshoot it

The diagnostic order:

1. Is the SDK initialised? Inspect the application log for the init line. If it is absent, the bug is in the bootstrap code that creates the TracerProvider. The fix is to call the builder; nothing else works without it.
2. Is the collector receiving anything? Scrape otelcol_receiver_accepted_spans. If it is zero, the failure is between the application and the collector. Check DNS, NetworkPolicy, and the protocol configuration.
3. Is the collector dropping? Scrape otelcol_exporter_dropped_spans. If it is non-zero, the queue overflowed. Either the application is producing faster than the exporter can drain (raise num_consumers or queue_size), or Tempo is slow (check tempo_ingester_traces_created_total).
4. Is the exporter failing? Scrape otelcol_exporter_send_failed_spans. If it is non-zero, the collector is reaching Tempo and Tempo is rejecting. Check Tempo’s logs and its ingester health endpoint.
5. Is Tempo receiving? Use tctl trace search or query the Tempo HTTP API directly. If Tempo has the trace ID but with fewer spans than expected, the failure is upstream of the collector (a code path that did not create a span).
6. Is the application version correct? Confirm the running image hash matches the expected one. A staging or debug build may have telemetry stripped.

Security implications

The diagnostic ladder does not require elevated credentials in most setups. The collector’s metrics endpoint on :8888 should be bound to the cluster network only, not exposed externally. The Tempo HTTP API behind the search and show endpoints accepts read-only credentials at most; trace payloads may contain PII or session tokens that the application stamped on the span.

The second-order risk is around the OTel endpoint URL. An attacker who can write to the application’s environment can re-point OTEL_EXPORTER_OTLP_ENDPOINT to a host they control and capture every span the application emits. Bind the application identity to a known collector address, and validate the endpoint URL at boot.

Performance implications

The cost of a complete trace is roughly 1 KB per span on the wire, plus the cost of the SDK creating the span in memory and serialising it. The batch processor amortises the cost by sending many spans per request.

• No-op SDK. When the SDK is initialised with no exporter, the span is created in memory and discarded. The cost is roughly 200 ns per span. This is what happens when auto-instrumentation is enabled but no TracerProvider is set.
• Buffered SDK. When the batch processor is configured, the span lives in the queue for up to timeout (default 5 s) before it is sent. Memory is proportional to max_queue_size times the average span size. At max_queue_size: 2048 and 1 KB per span, the queue is roughly 2 MB.
• Saturation. When the exporter is slower than the producer, the queue fills. The drop policy is to drop the oldest spans first. The visible effect is a Tempo picture that undercounts recent activity, which biases the latency histogram downward.

The right sizing depends on the per-service span rate and the network path to the collector. The metric otelcol_exporter_queue_size should stay below 80 percent of queue_size in steady state.

Production guidance

• Expose the collector’s own metrics. service.telemetry.metrics on a Prometheus scrape endpoint is the only way to see the drop counter, the queue size, and the send failure rate. Without it, span loss is silent.
• Alert on dropped_spans. The threshold is zero. A non-zero counter means the system is losing evidence. Page on the rate of change, not on the absolute count.
• Match receiver protocols. The OTLP/gRPC and OTLP/HTTP receivers are distinct components. Pick one per port. A common production bug is configuring the application for gRPC while the collector only listens for HTTP.
• Right-size the batch processor. Defaults are tuned for thousand-span-per-second services, not for high-volume services. The right value depends on the network round-trip to Tempo and the per-flush overhead. Benchmark before assuming the defaults.

Verification

You should now be able to answer:

• What is the difference between a span that was never generated and a span that was generated and dropped in transit?
• What is the diagnostic order when a trace has fewer spans than expected?
• Which collector self-metric indicates that spans are being dropped by the exporter queue?
• What is the most common production cause of missing spans?

Quiz