Broken Trace Propagation

The on-call engineer has a failed checkout from 03:14. The trace in Tempo has four spans: API gateway, inventory, pricing, payment. The trace_id is 0af7651916cd43dd8448eb211c80319c. The on-call engineer runs the same query against the warehouse service’s spans. The warehouse spans are there, with a parent span for the picklist generation, but the trace_id is different — b7ad6b7169203331e4f1b8a9c5e3f4d2. The inventory service documentedly calls the warehouse service to reserve stock. The trace says it did not happen. Or, more precisely, the trace says the call happened twice — once with one trace_id and once with another — and the warehouse span cannot be linked to the inventory span.

This is the lesson. The trace_id changed mid-request. The propagation broke at exactly one hop.

What it is

Trace propagation is the act of carrying the trace_id and the parent span_id from one service to the next across a request boundary. The W3C Trace Context standard defines the on-wire format; the OpenTelemetry SDK reads and writes it automatically when the propagator is configured.

A broken propagation is the failure mode where the receiving service either does not see the incoming traceparent header, or sees it but ignores it. The receiving service generates a fresh trace_id. The trace splits into two unrelated traces. The chain is broken.

The symptom is observable: two Tempo traces share an architectural link that no longer exists in the data. The user made one request; Tempo recorded two unrelated stories.

Why a sysadmin cares

The cost of a broken propagation is silent. The application keeps working. The user keeps getting responses. Tempo keeps storing spans. The chain that connects them is the only thing missing.

Three operational payoffs ride on propagation:

1. End-to-end latency. A trace that loses the chain cannot answer the question “what was the total latency from edge to warehouse?” The warehouse latency is in trace B. The everything-else latency is in trace A. The number that matters does not exist.
2. Failure attribution. When the inventory service fails to reserve stock, the team needs to know whether the failure was the warehouse’s fault or the network’s fault. The two traces cannot be compared because they have different trace_ids.
3. Trace-to-logs pivot. Grafana 11.x uses the trace_id from a Tempo trace to filter Loki logs. A broken propagation means the pivot works for half the services and is silently useless for the rest.

The diagnostic is mechanical. The cause is structural.

How it works — the mental model

The chain is carried by one HTTP header on every request between services.

Service A
  +-- creates trace_id 0af7...
  +-- creates span_id  b7ad...
  +-- outbound HTTP call
       traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
                          |                                |                |
                          |                                |                +-- flags (01)
                          |                                +-- parent span_id
                          +-- trace_id
       |
       v
Service B
  +-- reads traceparent, extracts trace_id and parent span_id
  +-- creates its own span_id, marks parent_span_id = b7ad6b7169203331
  +-- outbound HTTP call (propagates 0af7... + its own span_id)
       |
       v
Service C
  +-- reads, extracts, propagates again

The chain works because the SDK is configured to honour the header on every inbound request and to write the header on every outbound request. A service that does not configure the propagator at all is the most common cause of a broken chain.

The chain breaks at the first service that generates a fresh trace_id instead of reading the incoming one. Every downstream hop continues from the new ID. The two chains look unrelated.

How to configure it

The OpenTelemetry SDK configuration is the place where the propagation contract is honoured or broken. Three lines, in three different languages, demonstrate the pattern.

// Go
import "go.opentelemetry.io/otel/propagation"

func main() {
    // Set the global propagator to W3C Trace Context.
    // Without this, the SDK reads no headers and writes none.
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))
}

# Python
from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap

set_global_textmap(TraceContextTextMapPropagator())

// Node.js
const { propagation, trace } = require('@opentelemetry/api');
const { CompositePropagator, W3CTraceContextPropagator } =
    require('@opentelemetry/core');

propagation.setGlobalPropagator(
    new CompositePropagator({
        propagators: [new W3CTraceContextPropagator()],
    })
);

The pattern is identical: set the global propagator to W3C Trace Context at process start. The auto-instrumentation libraries read and write the header automatically once the propagator is set.

The message-bus hop — Kafka, in this example — requires explicit code to copy the context from the producer side into the message header and back out on the consumer side. Auto-instrumentation covers HTTP and gRPC; it does not cover Kafka.

// Kafka producer — copy context into the message headers
import "go.opentelemetry.io/otel/propagation"

func produce(ctx context.Context, msg *sarama.ProducerMessage) {
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    for k, v := range carrier {
        msg.Headers = append(msg.Headers,
            sarama.RecordHeader{Key: []byte(k), Value: []byte(v)})
    }
    producer.Input() <- msg
}

// Kafka consumer — extract context from the message headers
func consume(msg *sarama.ConsumerMessage, handler func(context.Context, []byte)) {
    carrier := propagation.MapCarrier{}
    for _, h := range msg.Headers {
        carrier[string(h.Key)] = string(h.Value)
    }
    ctx := otel.GetTextMapPropagator().Extract(context.Background(), carrier)
    handler(ctx, msg.Value)
}

A service that produces without injecting or consumes without extracting breaks the chain. The fix is structural, not config-level.

How to validate it

The validation ladder:

# 1. Does an incoming request carry the traceparent header?
curl -s -D - https://api.example.com/checkout -o /dev/null \
    -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
    | grep -i traceparent
# X-Trace-Id: 0af7651916cd43dd8448eb211c80319c
# (the edge generated or honoured a trace ID and stamped it on the response)

# 2. Does the downstream service see the same trace_id?
curl -s https://inventory.example.com/reserve \
    -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
    | jq '.trace_id'
# "0af7651916cd43dd8448eb211c80319c"
# (matches the parent)

# 3. Does the Kafka hop carry it?
# Inspect the headers on a produced message:
kcat -b kafka:9092 -t picklist -C -o beginning -e -f '%h\n' | grep traceparent
# traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

# 4. Does Tempo show the chain intact?
tctl trace show 0af7651916cd43dd8448eb211c80319c | jq '.spans | length'
# 7 (or however many the architecture predicts)

# 5. Does the trace-to-logs pivot work?
# Open the trace in Grafana -> click "Logs for this trace" ->
# the pivot opens Loki filtered by trace_id=<the same value>.

# 6. Does the propagator read the right format?
grep -r "TraceContext\|B3\|Jaeger" /etc/otelcol-contrib/ config/
# (any non-W3C propagator on a fleet that speaks W3C is the bug)

The fourth command is the answer. If the trace_id is consistent across every span and the count matches the architecture, the propagation is intact. If the trace_id changes mid-tree, the ladder above localises the hop.

How it can fail

Six recurring failure modes.

1. The propagator is not set. The SDK initialises the TracerProvider but never calls SetTextMapPropagator. The SDK creates spans but neither reads incoming traceparent headers nor writes them on outbound calls. Symptom: every service has spans, every trace has only one span (the receiving service generated a new ID). Tempo is full of single-span traces.
2. A custom header was invented. The team uses X-Correlation-Id instead of traceparent. One service writes it, the next does not read it. Symptom: the X-Correlation-Id is preserved in logs (the team believes the chain works), but Tempo shows two unrelated traces.
3. The HTTP client strips the header. A corporate proxy or a service mesh sidecar is configured to strip traceparent. The service that makes the call is innocent. Symptom: every outbound call from one specific service starts a new trace. The chain breaks at one specific service boundary.
4. The async hop drops the context. A Kafka producer does not call propagator.Inject before sending. A cron job dispatches work without copying the context. Symptom: the work performed by the consumer has no parent span link to the request that scheduled it. The trace is split across the boundary.
5. Two propagators disagree. The producer uses W3C, the consumer uses B3. The receiving service extracts the wrong header and ignores the W3C one. Symptom: Tempo shows the W3C trace and the B3 trace as separate chains for the same request. Often follows a partial migration off Zipkin.
6. The trace_id is correct but the parent_span_id is reset. The downstream service reads the trace_id but treats the request as a new root (no parent). Symptom: Tempo shows one trace with two roots and no parent-child link. The chain is broken topologically even though the trace_id matches.

How to troubleshoot it

The diagnostic order:

1. Confirm the symptom. Pull two spans from Tempo that should share a trace_id. Run tctl trace show <id> on each. If the IDs differ, the propagation is broken at the boundary between them.
2. Inspect the wire at the boundary. curl -v with a known traceparent header, or tcpdump -A on the receiving port. Does the traceparent arrive?
3. Inspect the SDK configuration. Every service in the chain must set the global propagator at process start. A service that initialises the TracerProvider without setting the propagator is a propagation failure.
4. Inspect the message-bus producer and consumer. The producer must inject the context into the headers. The consumer must extract it. Neither is automatic for Kafka / RabbitMQ / SQS.
5. Inspect the service mesh / proxy. Istio and Envoy can be configured to strip headers. Confirm the mesh is honouring traceparent.
6. Diff the propagator formats. A partial migration off B3 or Jaeger will leave some services on the old format and some on the new. The chain breaks at the boundary.

Security implications

The traceparent header is not sensitive. It contains a 128-bit random ID and a 64-bit random ID, both opaque. The header is safe to log, safe to forward, and safe to store.

The risk is around header injection. A service that echoes an incoming header back to the client without validating the format is one step from a header-injection or request-smuggling vulnerability. The convention is to allow only well-formed traceparent values (regex: ^[0-9a-f]{2}-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$) and to generate a fresh header at the edge if the incoming one is malformed.

A second-order risk: a malicious client that supplies a traceparent header at the edge can pin a trace_id they control. The on-call engineer who later searches by trace_id may surface a trace the client constructed. The remediation is either to overwrite the incoming header at the edge with a freshly generated one, or to log the original alongside the overwritten value for traceability.

Performance implications

The cost of propagation is one header on every outbound HTTP call and roughly 100 bytes of overhead in the message envelope on async hops. At 10 000 requests per second, the header overhead is roughly 1 MB/s of egress on the wire. Measurable but not material.

The expensive path is reading the current span context and formatting the traceparent string. The cost is roughly 100 ns per call. At the same 10 000 requests per second, the CPU overhead is 1 ms per second — well under 1 percent of a single core.

The performance-relevant failure mode is the missing propagator. A service that never reads the incoming traceparent does not avoid work — it starts a new trace, which generates a new trace_id, which means every downstream service creates a new span with no parent. The latency histogram in Tempo becomes useless for that chain. The cost is paid in incident response, not in CPU cycles.

Production guidance

• Set the propagator at process start. Every service, every language, every binary. There is no per-request configuration; the propagator is global for the lifetime of the process.
• Use W3C Trace Context as the only format. Adopting a second format is a partial migration waiting to fail. Pick one and retire the other.
• Inject and extract at every async boundary. Kafka, RabbitMQ, SQS, cron, Celery, Sidekiq, the message bus, the in-process queue. None of them are covered by auto-instrumentation.
• Validate the mesh. Confirm that the service mesh or proxy is honouring traceparent and not stripping it.
• Alert on trace_id fragmentation. Tempo does not alert on this directly. A reasonable proxy is to alert on the rate of single-span traces per service. A service that emits mostly one-span traces is not propagating.

Verification

You should now be able to answer:

• What is the on-wire format of the traceparent header?
• How do you recognise a broken propagation from the Tempo UI?
• What is the difference between a missing propagator and a stripped header?
• Where does the propagation chain most often break?

Quiz