Parent / Child Relationships · Observability

The trace UI shows two services that obviously depend on each other, but every trace ends at the gateway. The backend service has zero traces. The on-call engineer suspects an exporter bug, redeploys the collector, and is no closer. The actual cause is a six-line configuration on the gateway that strips the traceparent header on HTTP/2 to HTTP/1.1 downgrade. The trace context was lost in flight. That is the most common production failure of parent / child relationships: the chain breaks at the seam where two services meet, and the engineer chasing the problem is looking in the wrong layer.

What it is

The parent / child relationship between spans is encoded by one field: parent_span_id on the child span. For the child to know its parent’s identifier, the parent must hand that identifier to the child before the child starts its own span. In a single process this is a thread-local; across processes it is a propagator — a serialised, transport-specific encoding of the trace context that travels with the request.

The W3C Trace Context specification defines a single wire-format that every compliant SDK, proxy, and middleware recognises: the traceparent and tracestate HTTP headers. Every other propagator (B3, Jaeger, X-Ray) is either legacy or vendor-specific; production deployments standardise on W3C.

Why a sysadmin cares

The parent / child relationship is what makes a collection of spans a trace. Without it, every span is an island. The operational pain shows up three ways:

Search by trace ID fails to span services. The user has a trace_id from the access log; Tempo returns the spans from the gateway and nothing else. The investigation dead-ends.
Latency is unattributable. The root span says “this request took 4 seconds” but every child belongs to a different service and the flame graph has no shared ancestor.
Fan-out is invisible. A single user request fans out to twelve services. Without propagation, the on-call engineer sees twelve independent traces and no way to know they belonged to one user action.

Propagation is the contract between services. If the contract is honoured, the platform is observable. If the contract is broken at any hop, the trace is broken from that hop onward.

How it works

Inside one process, the SDK keeps the current span in a context (SpanContext) stored in a thread-local (or equivalent — async frameworks use task-locals). When the application starts a new span, the SDK reads the current context and copies the trace_id and parent_span_id into the new span. When the application makes an outbound call, the SDK serialises the current context into headers, attaches them to the call, and the receiving service reverses the process.

The W3C traceparent header is 55 characters of plain ASCII:

traceparent: 00-<trace_id>-<parent_span_id>-<flags>

00           -- version (currently 0)
4bf92f3577b34da6a3ce929d0e0e4736   -- 16-byte trace_id, lowercase hex
00f067aa0ba902b7                   -- 8-byte span_id of the caller
01                                -- flags: 01 = sampled, 00 = not sampled

The tracestate header carries vendor-specific data and is optional; the traceparent is the contract. Every compliant library must accept and emit it; nothing else is required for parent / child propagation to work across HTTP boundaries.

A typical request flow:

client              gateway             checkout            payment
  | -- traceparent ----> |                   |                   |
  |   00-aaaa...-bbbb...-01 |                   |                   |
  |                        | -- traceparent --> |                   |
  |                        |   00-aaaa...-cccc...-01               |
  |                        |                   | -- traceparent --> |
  |                        |                   |   00-aaaa...-dddd...-01 |
  |                        |                   |                   |
  v                        v                   v                   v
root span                edge span           server span          client span
trace_id=aaaa...         trace_id=aaaa...    trace_id=aaaa...    trace_id=aaaa...
parent_span_id=0000...   parent_span_id=bbbb  parent_span_id=cccc  parent_span_id=dddd
                                              (parent of payment)

All four spans share trace_id=aaaa.... The parent’s span_id appears as the child’s parent_span_id. The chain is intact.

Under the hood

What the current span means in an async runtime

The OpenTelemetry SDK tracks the “current” span through a Context stored in the language’s per-task state. In Go this is context.Context; in Python it is contextvars; in Java it is a ThreadLocal for sync code and an Executor instrumentation for async. In Node.js and asyncio it is a task-local.

Three places break this in practice:

Background workers. A span started on the request thread is not in scope on the worker thread. The worker creates a new root span by default. Symptom: the worker traces look unrelated to the request traces.
Fire-and-forget queues. The producer publishes a message with the current span context in headers; the consumer must extract it. If the consumer’s SDK is not configured to read the propagation headers, every consumer trace is a new root.
Retries. A retry of the same logical operation produces a new span. Without an explicit span link from the original to the retry, the chain has a fork the analyst has to reconstruct by hand.

The fix in every case is explicit: extract the parent context from the inbound carrier, attach it as the parent of the new span, run the work inside the new span. There is no shortcut.

How to configure it

Application side — Python, registering the W3C TraceContext propagator explicitly so it does not silently fall back to a vendor default:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagators.b3 import B3Format
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Explicit propagator list. W3C first, B3 for one legacy downstream.
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)

# propagators are global; setting them here means every instrumentation
# library respects them.
from opentelemetry import propagate
propagate.set_global_textmap(
    TraceContextTextMapPropagator()  # W3C only -- drop B3 unless a legacy caller needs it
)

tracer = trace.get_tracer("checkout")

with tracer.start_as_current_span("POST /checkout") as root:
    # Inject context into an outbound HTTP call manually if no
    # auto-instrumentation is doing it.
    headers = {}
    propagate.inject(headers)
    # headers now contains "traceparent: 00-..."
    requests.post("http://payment-svc/charge", json={...}, headers=headers)

Collector side — the collector itself does not propagate context, but it can rewrite or strip headers if asked. The default behaviour is to leave headers alone. If a corporate proxy strips traceparent on HTTP/1.1 fall-back, the fix is at the proxy, not at the collector.

Grafana Alloy side — Alloy has no propagation role. It receives OTLP from the application SDK and forwards it to Tempo.

How to validate it

The simplest test: send a request through two services and inspect both halves of the trace. They must share the trace_id.

# 1. Generate a request that hits the gateway and the backend.
TRACE=$(curl -sv -X POST http://gateway.internal/checkout \
  -H 'Content-Type: application/json' \
  -d '{"cart_id":42}' 2>&1 \
  | grep -i '^< traceparent' | awk '{print $3}' | cut -d- -f2)

echo "trace_id=$TRACE"

# 2. The gateway must have emitted a SERVER span.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode "q={ service.name = \"gateway\" && trace_id = \"$TRACE\" }" \
  http://tempo.internal:3200/api/search | jq '.traces | length'
# Expected: 1

# 3. The backend must also have emitted a span with the SAME trace_id.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode "q={ service.name = \"checkout\" && trace_id = \"$TRACE\" }" \
  http://tempo.internal:3200/api/search | jq '.traces | length'
# Expected: 1

# 4. The full trace must show a parent-child link.
curl -s -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].scopeSpans[].spans[]
         | {name, service: .attributes[]? | select(.key=="service.name") | .value.stringValue}'
# Expected: at least two spans, both with the same trace_id,
# one named after the gateway route, one named after the backend operation,
# and the backend span must have parent_span_id equal to the gateway span_id.

If step 3 returns 0 while step 2 returns 1, the trace context is being lost at the gateway. The diagnostic is to capture the raw HTTP exchange:

# Capture the actual headers on the wire.
mitmproxy --mode reverse:http://backend.internal:80 \
  --set flow_detail=3

# Or with tcpdump on the loopback interface.
tcpdump -i lo -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' \
  | grep -i traceparent

The captured payload must contain traceparent: 00-... between the two services. If it does not, the gateway is the culprit; find the rewrite rule.

How it can fail

Header stripped at the gateway. A reverse proxy, load balancer, or CDN rewrite rule deletes the traceparent header on protocol downgrade, response header sanitisation, or CORS preflight handling. Symptom: the gateway emits a trace, the backend does not; every backend trace is a new root.
Mixed propagators. One service emits W3C, another expects B3, the SDK silently does not extract. Symptom: every other request propagates correctly and the failure looks like noise.
Async boundary loss. A span is created on the request thread but the work runs on a worker pool without context propagation. Symptom: traces from background workers have no parent and a fresh trace_id each time.
Queue producer/consumer mismatch. The producer injects traceparent into message headers; the consumer does not extract it. Symptom: queue-driven traces look unrelated to the producer traces.
Retry as new root. A retry of the same logical call emits a fresh span with no link to the original. Symptom: the same operation shows up twice in Tempo with different trace IDs and no obvious relationship.
Baggage dropped. Application-level correlation IDs travel in baggage (or in custom headers) and are stripped by a different layer than traceparent. Symptom: the trace is intact but the user-facing correlation ID disappears between services.

How to troubleshoot it

What if the trace breaks exactly between two services?

Check the wire. tcpdump or mitmproxy on the receiving side. The inbound request must carry a traceparent header. If it does not, the upstream service is failing to inject, or the network path is stripping it.
Check the receiving SDK. Confirm the receiving service registers the same propagator. Run propagate.get_global_textmap() (or the equivalent for the language) — the result must be a W3C TraceContext propagator or a composite list that includes it.
Check the proxy. Most reverse proxies expose a config knob to forward unknown headers. NGINX needs proxy_pass_request_headers on; Envoy needs the header listed in request_headers_timeout allow-lists. A default “strip unknown headers” rule is the most common cause.
Check for downgrades. HTTP/2 to HTTP/1.1 fallback in some load balancers drops headers with underscores or with non-standard casing. traceparent is fine; TraceParent with mixed case is not.
Add an explicit test. A minimal client that issues a request with a known traceparent and a server that asserts on it isolates the network from the SDK.

Security implications

The traceparent header is a 128-bit identifier; it is not a secret. It can safely cross trust boundaries. What it can do is leak topology: an external client that can read its own traceparent can infer the existence of an internal service boundary if it can correlate trace IDs with response timing. For B2B APIs that expose the header back to the caller, this is expected. For internal services, do not echo traceparent back to the client unless that is the explicit design.

The tracestate header carries vendor data and is the one to watch: malformed tracestate can be used to inject false context. The W3C spec mandates that compliant SDKs reject malformed tracestate values; verify this in the receiving library before relying on it.

Performance implications

Header injection is two string concatenations per outbound call. Header extraction is a parse of 55 bytes per inbound call. Both are negligible compared to the application work they describe. The cost of broken propagation, however, is paid at incident time: every trace that should have been one trace becomes many, every “which dependency” question reverts to guesswork, every minute of mean time to resolution rises.

Production guidance

Test propagation as part of CI. A test that issues a request through two services and asserts that both halves share a trace_id catches the most common production failure.
Document the propagator choice in the platform runbook. “W3C TraceContext, no exceptions” is a one-line answer that prevents a year of “which propagator should we use” arguments.
For queues, standardise on a header name (often traceparent) and document it in the team SDK README.

Verification

You should now be able to answer:

What do the four fields of the W3C traceparent header represent?
How does the SDK know which span is the parent of a new span started in an inbound HTTP handler?
What is the typical production cause of a trace that ends at the gateway with no backend spans?
How do you confirm at the wire that the receiving service is receiving a valid traceparent header?
Why is a queue consumer likely to start a new trace if you do not configure message-header extraction explicitly?