Context Propagation

A checkout request arrives at the API gateway. The gateway records a span. The gateway calls the order service, which records a span. The order service calls the payment service, which records a span. The payment service calls the fraud service, which records a span. In Tempo, the four spans appear as four separate traces with four different trace IDs.

The trace is broken. The spans are correct. The gap is at the boundary between the gateway and the order service, where the parent’s trace ID was not injected into the downstream call’s HTTP header. The lesson is not about the SDK. It is about the headers that cross the boundary, and the protocol-level contracts that have to be honoured at every hop.

What it is

Context propagation is the discipline of moving the trace context — the trace ID, the parent span ID, the trace flags, and any vendor-specific baggage — across a process or network boundary. The OpenTelemetry specification defaults to W3C Trace Context, which carries the context in two HTTP headers:

• traceparent — the version, the trace ID, the parent span ID, and the trace flags. The canonical shape:

  00-<32-hex trace-id>-<16-hex parent-id>-<2-hex flags>
  00-a0892f3577b34da6a3ce929d0e0e4736-f03067aa0ba902b7-01

• tracestate — vendor-specific key-value pairs. The OpenTelemetry project uses this for ot= entries that describe the trace’s path through the system. The header is opaque to receivers and the rules for editing it are narrow.

The same context is carried over gRPC via the grpc-trace-bin header (the binary form of the trace context) and over Kafka and other messaging protocols via the message header. The content is the same; the carrier is different.

+--------+      traceparent       +--------+
| Span A |  ------------------->  | Span B |
| (root) |  tracestate           | (child)|
+--------+                        +--------+
   service.inbound service.outbound
                 |
                 v
        W3C Trace Context spec

The context is set at the start of the span and read at the start of the next. The two operations are inject (write to the carrier) and extract (read from the carrier). The OpenTelemetry SDK runs them automatically when the auto-instrumented HTTP / gRPC / messaging library is in use.

Why a sysadmin cares

A trace that breaks at the first hop is a collection of isolated spans. Tempo shows the user journey as four traces instead of one. The on-call engineer has to manually correlate them by user.id and timestamp, which is exactly the kind of detective work the trace was supposed to remove.

The boundary that breaks is rarely the application’s. It is the proxy that strips headers, the message queue that does not preserve them, the async task that loses them when it crosses the thread boundary, the CDN that resets them, the sidecar that rewrites them. The lesson is “the boundary is the contract”, and the contract has to be enforced at every hop, not just the one the developer is debugging.

How it works

The mental model has three layers and one rule.

+---------------------------+
| Application Span          |
| (SpanContext: traceID,    |
|  spanID, flags)           |
+---------------------------+
             |
             | inject (write to carrier)
             v
+---------------------------+
| Carrier (HTTP header,     |
|  gRPC metadata, message   |
|  header, queue payload)   |
+---------------------------+
             |
             | extract (read from carrier)
             v
+---------------------------+
| Application Span          |
| (new SpanContext, parent  |
|  = previous spanID)       |
+---------------------------+

The rule is that the parent span ID at the next hop equals the span ID at the previous hop. The continuity is what Tempo uses to join the spans into a single trace.

How to configure it

The propagation chain is configured once at SDK bootstrap. The default is W3C TraceContext + W3C Baggage, which is the right production starting point.

Python.

from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Default: W3C TraceContext + Baggage is already configured.
# Override only when the deployment calls for it.
set_global_textmap(TraceContextTextMapPropagator())

Go. The propagator is set on the TracerProvider’s TextMapPropagator field.

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
)

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

The application code does not need to know about the propagator. The auto-instrumented HTTP client and server pick up the global propagator and call inject and extract on every request.

Manual propagation. When the boundary is not HTTP (in-process queue, custom protocol, async task), the developer has to call inject and extract explicitly.

from opentelemetry import trace, context as otel_context
from opentelemetry.propagate import inject, extract

# In the producer
ctx = otel_context.get_current()
headers = {}
inject(headers)
queue.put({"payload": ..., "_otel": headers})

# In the consumer
message = queue.get()
ctx = extract(message["_otel"])
token = otel_context.attach(ctx)
with trace.get_tracer("worker").start_as_current_span("process"):
    ...
otel_context.detach(token)

This is the failure shape. The producer / consumer call inject / extract correctly, but the worker starts the span before calling attach, so the parent is missed. The fix is to attach first and start the span inside the attached context.

How to validate it

The validation is to confirm that the traceparent header is on the request and that the trace ID matches.

curl -sv -H "traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"    http://checkout-svc:8080/api/v1/orders 2>&1 | grep -iE 'traceparent|< http'

TRACEID=$(uuidgen | tr -d '-')
curl -s -H "traceparent: 00-$TRACEID-aaaaaaaaaaaaaaaaaa-01"    http://gateway/api/v1/orders
tempo-cli query "{ trace = "$TRACEID" }"

$ curl -sv -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' http://checkout-svc:8080/healthz 2>&1 | grep -iE 'traceparent'

> traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
< traceparent: 00-0af7651916cd43dd8448eb211c80319c-1f2e3d4c5b6a0987-01

# Receiver uses B3, sender uses W3C: nothing in the B3 header, trace breaks.
# Receiver uses W3C, sender uses B3: B3 header arrives but W3C parser sees no traceparent.

# The audit:
curl -s http://payment-svc:8080/admin/propagators | head
curl -s http://checkout-svc:8080/admin/propagators | head

How it can fail

Six failure shapes recur. The first four are misconfigurations on the boundary; the last two are operations that lose the context in flight.

1. The proxy strips the traceparent header. The Cloudflare or AWS ALB or corporate proxy is configured to drop unknown headers. The downstream service receives no context and starts a new trace. Symptom: every request produces two traces at the boundary, with the downstream trace having no upstream span in Tempo.
2. The server rewrites the trace ID. A misconfigured sampler or load test harness generates a fresh trace ID per request. The trace is correct per-service but is never joined. Symptom: traces are short; the upstream span never appears.
3. The async queue loses the context. The producer publishes a message; the consumer reads the message and starts a span. The trace context is not on the message header. Symptom: the consumer’s trace shows a root span with no parent; the producer’s trace shows a span that ends before the queue.
4. The thread boundary loses the context. The handler receives the request, hands work to a thread pool, and the worker function runs without the propagated context. The worker starts a new span as a root. Symptom: the trace breaks at the asynchronous hop.
5. The propagator is mismatched between services. One service emits B3 headers (legacy), the other expects W3C. The trace is broken at this hop. Symptom: the W3C service shows no traceparent; the B3 service shows no b3 header.
6. The context is dropped on a timeout retry. The client times out, the retry layer starts a new context, and the original trace is left without the retry’s span. Symptom: the trace is missing the retry span; the original span ends with an error.

How to troubleshoot it

The diagnostic order is “is the carrier on the wire?”, “is the receiver extracting it?”, “is the parent joining?”, “is the trace joined in Tempo?”.

curl -sv http://checkout-svc:8080/api/v1/orders 2>&1 | grep -iE 'traceparent'

curl -sv http://checkout-svc:8080/api/v1/orders 2>&1 | grep -iE '< traceparent'

tempo-cli query '{ trace = "0af7651916cd43dd8448eb211c80319c" }'

Security implications

The traceparent header carries the trace ID and the parent span ID. The trace ID is a 16-byte random value; the span ID is an 8-byte random value. Neither is sensitive by itself. The tracestate header carries vendor-specific key-value pairs that the application controls; the OTel spec is explicit that the values should not contain PII or credentials.

The W3C Baggage header is the one that can leak. The header is designed to carry arbitrary key-value pairs across services, and the natural temptation is to put the user ID, the tenant ID, or the trace ID in there. The mitigation is the same as for attributes: scrub at the SDK boundary, and never put credentials in baggage.

The OTel documentation gives a security recommendation that is worth restating: be cautious when accepting context from external sources. The propagator can be configured to ignore the incoming trace context and start a new trace per request, which is the right answer for a service that sits on the public edge.

Performance implications

The header is 70-110 bytes on the wire. The injection and extraction are sub-millisecond and use no allocations beyond the header map. The performance cost is negligible.

The cost is in the trace backend. A trace with 1000 hops where every hop is correctly joined produces 1000 spans under one trace ID. The backend’s index for that trace ID is proportional to the number of spans. The discipline is to keep hops meaningful (do not propagate across unrelated services) and to use sampling to bound the trace volume (the next lesson in the series).

Production guidance

Verification

You should now be able to answer:

• What is the format of the traceparent header, and what does each field mean?
• Why is the propagation contract dependent on the middlebox configuration and not on the application’s code?
• What is the failure shape you would expect from a producer that publishes a message to Kafka without injecting the trace context?
• Why does the worker thread that consumes the message need to attach the context before starting the span?
• Why is the tracestate header considered opaque to receivers?

Quiz