Propagation Broken

A trace lookup returns trace not found. The on-call engineer follows link A and link C of the missing-trace chain and confirms both are healthy. The SDK is running; the collector is receiving spans from the payment service and from the database service. But the trace the user is on is not in Tempo. The application log for the payment service shows a trace_id=4bf92f3577b34da6. The application log for the database service, called one millisecond later by the payment service, shows a different trace_id. Two traces in Tempo, both with one span, neither with the full journey. The W3C traceparent header did not survive the boundary between payment-svc and database-svc. Link B is broken.

What it is

Propagation is the act of carrying the trace context across a service boundary. The OpenTelemetry SDK uses the W3C Trace Context specification, which defines two HTTP headers:

  traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              |  |                                          |               |
              |  |                                          |               | flags: 01 = sampled
              |  |                                          | span-id: 16 hex
              |  trace-id: 32 hex
              version: 00

  tracestate: vendor1=value1,vendor2=value2

A propagation propagator on the client side injects the header on the outbound request. A propagation propagator on the server side extracts the header on the inbound request and starts a child span. If either side fails, the trace is split.

Link B is broken when:

• The header is not injected on the outbound call.
• The header is injected but stripped by an intermediary (proxy, sidecar, load balancer).
• The header is injected and passed but ignored by the receiver (the SDK is configured with no extractors).
• The header is passed but with a non-W3C format (B3, Jaeger, X-Ray) that the receiver does not recognise.

The visible shape at Tempo: two separate traces for what is logically one request. The application logs on each side show trace IDs, but the IDs differ. Tempo cannot stitch the two halves because they have no shared trace ID.

Why a sysadmin cares

Broken propagation is the silent failure shape of a distributed trace. The SDK is healthy; the collector is healthy; the exporter is healthy. Every metric on the collector says “we are receiving spans”. But the trace is not the trace the user is on. Investigation time rises because the operator opens the wrong trace ID at Tempo and sees one isolated span, then opens another trace ID and sees another isolated span. The minutes spent reconciling the two are the cost of link B.

The diagnostic reflex is “look at the boundary”. The boundary is where the header should appear and may not. The cheapest diagnostic that catches link B is a tcpdump or a curl -v against the boundary. If the header is absent on either side, the cause is at the boundary; if the header is present on both sides but the trace IDs differ, the cause is in one of the propagators.

How it works

Propagation operates on every outbound and inbound request at the boundary:

  +-----------------+                 +-------------------+
  |  Service A      |                 |  Service B        |
  |                 |  -- HTTP/gRPC -->|                   |
  |  propagator     |                 |  propagator       |
  |  inject(...)    |                 |  extract(...)     |
  |                 |                 |                   |
  |  span (parent)  |                 |  span (child)     |
  +-----------------+                 +-------------------+
         |                                    |
         |    +--------------------------+    |
         +--> | proxy / sidecar / LB     | <--+
              |                          |
              |  allowlist / denylist    |
              |  on header rewrite       |
              +--------------------------+

The propagator on the client side is a small piece of code that reads the current span context from the SDK and writes the W3C headers to the outbound request. The propagator on the server side reads the W3C headers and starts a new span as a child of the incoming context.

A proxy in the middle can rewrite the header, strip it, or leave it. If it strips it, both ends still work; the trace is split. If it rewrites it to a non-W3C form, the receiver sees an unknown header; the SDK extracts nothing; the new span starts with a fresh trace ID.

The B3 and Jaeger legacy

Before W3C Trace Context was a recommendation, Jaeger defined a similar but distinct header set:

  # Jaeger format (legacy)
  uber-trace-id: 4bf92f3577b34da6a3ce929d0e0e4736:00f067aa0ba902b7:0:1
  uber-trace-id-style: sw8

  # B3 (Zipkin) format
  X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
  X-B3-SpanId: 00f067aa0ba902b7
  X-B3-Sampled: 1

A service that emits B3 headers and a service that only extracts W3C headers do not propagate. The collector can bridge the two with a jaeger or zipkin receiver that normalises to W3C, but only if both sides are configured.

Auto-instrumentation and propagation

Auto-instrumentation handles propagation for the protocols it instruments. The Java agent instruments HTTP client classes and injects W3C headers automatically. The agent instruments HTTP server classes and extracts W3C headers automatically. If the agent is attached to the JVM, the propagation is automatic for the protocols the agent recognises. A custom HTTP client written with raw sockets does not get auto-instrumentation; the developer must propagate by hand. That custom client is the most common link-B failure in mixed codebases.

How to configure it

Propagation is configured in two places: the SDK bootstrap and the collector’s receiver.

SDK propagation list

  // Go SDK
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
      propagation.TraceContext{},
      propagation.Baggage{},
      // Optional: legacy interop
      // jaegerpropagator.Jaeger{},
      // b3propagator.B3{InjectEncoding: b3propagator.B3MultipleHeader},
  ))

  // Java SDK
  SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
      .setPropagators(TextMapPropagator.composite(
          W3CTraceContextPropagator.getInstance(),
          BaggagePropagator.getInstance()))
      .build();

Severity: CONFIGURATION. Restart the application after changing the propagator list.

Auto-instrumentation default

  # Java: W3C is the default; no extra config needed.
  ENV OTEL_PROPAGATORS=tracecontext,baggage

  # Python: W3C is the default.
  ENV OTEL_PROPAGATORS=tracecontext,baggage

  # Node: W3C is the default.
  ENV OTEL_PROPAGATORS=tracecontext,baggage

  # Add B3 for legacy interop:
  ENV OTEL_PROPAGATORS=tracecontext,baggage,b3

Severity: CONFIGURATION. Re-deploy the application.

Service mesh header allow-list

For Istio / Envoy, the header propagation for tracing is controlled by rbac and proxy_http filters. The default Envoy behaviour is to forward all headers; an explicit request_headers_timeout or a Lua filter can strip headers. For Linkerd, the default is forward-all; a ServiceProfile does not affect headers. For NGINX, the proxy_pass_request_headers directive is on by default.

  # Istio: do not filter traceparent. The default is correct.
  apiVersion: security.istio.io/v1beta1
  kind: RequestAuthentication
  metadata:
    name: default
  spec:
    jwtRules: []
  # No header rewriting required.

Severity: CONFIGURATION. Apply the manifest; Envoy picks it up on next push.

Collector dual-stack (B3 + W3C)

  # /etc/otelcol-contrib/config.yaml
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
    zipkin:
      endpoint: 0.0.0.0:9411

  connectors:
    spanmetrics: {}

  service:
    pipelines:
      traces:
        receivers: [otlp, zipkin]
        processors: [batch]
        exporters: [otlp/tempo]

  exporters:
    otlp/tempo:
      endpoint: tempo:4317
      tls:
        insecure: true

Severity: CONFIGURATION. Restart the collector.

How to validate it

Severity: READ-ONLY.

Inspect the boundary

  # Outbound: from payment-svc to database-svc
  kubectl exec deploy/payment-svc -- \
    curl -v http://database-svc:5432/healthz 2>&1 | \
    grep -iE 'traceparent|tracestate'
  # < traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
  # < tracestate: vendor1=value1

A missing traceparent on the outbound call is link B from the sender.

  # Inbound: server side, with tcpdump inside the database-svc pod
  kubectl exec deploy/database-svc -- \
    tcpdump -A -s0 -i any 'tcp port 5432' 2>&1 | \
    grep -iE 'traceparent|tracestate'
  # 14:23:18  traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

A present header inside the receiver confirms the inbound side received it.

Confirm the SDK extracted the trace context

  # Check the SDK self-observability counter for spans started
  # with the propagation flag set
  curl -s http://database-svc:9464/metrics | \
    grep -E '^otel_sdk_span_started.*parent='
  # otel_sdk_span_started_count{parent="remote_parent"} 8421

A flat parent="remote_parent" counter confirms the receiver never saw a parent context; the spans it created were all local.

Confirm the collector’s exporter sees both halves

  # Look at the trace_id field on received spans
  curl -s http://otel-collector:8889/metrics | \
    grep -E '^otelcol_receiver_accepted_spans'
  # otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 12842

  # Then query Tempo for both halves of the broken trace
  for tid in 4bf92f3577b34da6a3ce929d0e0e4736 7a3b...; do
    curl -s "http://tempo:3200/api/traces/$tid" | \
      jq '.resourceSpans[].scopeSpans[].spans[].name'
  done

Two separate trace IDs at Tempo with one span each is the classic link-B shape.

Confirm a service mesh allow-list is not stripping

  # Istio / Envoy: inspect the proxy config
  istioctl proxy-config routes deploy/payment-svc.cluster.local
  # Look for header_rewrite / request_headers_filters

  # NGINX: confirm proxy_pass_request_headers is on
  kubectl exec deploy/nginx -- \
    nginx -T 2>/dev/null | grep proxy_pass_request_headers
  # proxy_pass_request_headers on;

How it can fail

Six failure shapes, ordered by frequency in production fleets:

1. Service mesh sidecar with a header allow-list. The sidecar strips traceparent because the allow-list is populated by an audit-driven hardening change that did not include tracing headers. Symptom: traces are present on each side of the sidecar; the trace IDs differ; Tempo shows two traces.

2. Custom HTTP client without auto-instrumentation. The service uses an HTTP client that is not on the auto-instrumentation list (a third-party SDK, raw sockets, a vendored library). The developer did not propagate the trace context by hand. Symptom: trace breaks at the boundary between the custom client and the next service.

3. Receiver SDK configured without the right propagator. The receiver has W3C enabled but a B3 client sends the inbound request. The receiver extracts nothing; the trace is split. Symptom: spans are emitted on both sides but the trace IDs differ.

4. Two protocols stacked: W3C on one side, Jaeger on the other. A legacy Jaeger client emits uber-trace-id; the new service only extracts traceparent. The Jaeger client creates spans with no parent; the new service receives an unknown header and creates a child with a fresh trace ID. Symptom: trace IDs differ; Tempo shows the Jaeger half and the W3C half as separate traces.

5. Outbound call across an async queue. A message queue (Kafka, RabbitMQ, SQS) carries the message but not the headers. The producer injects the header on the HTTP publish call; the consumer reads the message body and never sees the traceparent. Symptom: the HTTP-publish trace and the message-consume trace are separate; the user-visible flow is broken.

6. traceparent header value malformed. A buggy proxy or a debug tool rewrites the header to an invalid value (wrong length, non-hex characters). The receiver’s parser rejects it; the receiver starts a fresh trace. Symptom: a small fraction of traces are split, with no pattern by service or region.

How to troubleshoot it

The diagnostic order, link B first, cheapest signal first:

1. Inspect the boundary. Pick a service that should be downstream of another service. kubectl exec into the downstream service and tcpdump for the traceparent header on the inbound port. If absent, the boundary is the cause.
2. Inspect the outbound side. kubectl exec into the upstream service and curl -v against the downstream. Confirm traceparent is present on the outbound call.
3. Confirm SDK extraction. Scrape the downstream SDK’s self-observability metrics. A flat parent="remote_parent" counter confirms the SDK never saw a parent context.
4. Confirm the SDK propagator list. The bootstrap config must list every protocol the boundary speaks. A W3C-only list and a B3-emitting upstream is a broken propagation.
5. Confirm the mesh sidecar. istioctl proxy-config or linkerd check. Look for header rewriting rules.
6. Look for trace ID pairs. Query Tempo for two trace IDs that should be one. The pattern of pairs across many requests confirms link B.

Security implications

Propagation headers carry the trace ID, the span ID, and a flag. None of these are sensitive on their own. The header is not an authentication credential. The header is not a session token. The header is metadata, and the metadata alone is not a confidentiality concern.

The boundary is the security concern. A mesh sidecar that forwards every header is also forwarding every credential. The allow-list pattern is correct; the implementation must include the trace headers explicitly. The fix is not to disable the allow-list; the fix is to add traceparent and tracestate to it.

A second concern is cross-tenant leakage via tracestate. The tracestate value is a vendor-specific opaque string. A misconfigured tenant boundary can carry tracestate across tenants, leaking the upstream vendor context to the downstream. The collector’s attributes or resource processor should clear tracestate at the tenant boundary.

Performance implications

Propagation is cheap. The W3C header is roughly 60 bytes on the wire; the B3 multi-header is roughly 100 bytes. The inject / extract logic is two string concatenations and a parse. The CPU overhead is well below 1% of the request budget at any sane QPS.

The bigger performance trap is the SDK’s baggage propagation, which is enabled by default and copies every baggage key/value pair on every outbound call. An application that places large values in baggage grows the header on every hop. The fix is to scope baggage to the keys that need cross-service propagation and to size-limit the values.

Production guidance

• Always run W3C Trace Context as the default propagator. Add B3 or Jaeger only for documented legacy interop.
• Always confirm traceparent is on the sidecar’s allow-list. Run a smoke test in CI that asserts the header is present on every inter-service call.
• Always scrape the SDK’s parent="remote_parent" counter and alert on a drop. The counter is the canary for link B.
• Always check the collector’s attributes processor for tracestate clearance. Cross-tenant leakage is silent.

Verification

You should now be able to answer:

• What is the W3C traceparent header format?
• Which header is the visible signal of intact propagation?
• What is the classic shape of a broken propagation in Tempo?
• Why does a service mesh sidecar with a header allow-list break propagation?
• How do you confirm the receiver SDK extracted the parent context?

Quiz