Trace Propagation Test

A team rolls out a new authentication proxy in front of every service. The proxy strips incoming headers that are not on an allow-list. The W3C traceparent header is not on the allow-list. The next morning the trace pipeline shows every service emitting one-span traces; the trace tree stops at the first hop. The metric latency is unchanged; the trace investigation has become impossible. The team finds the bug in forty minutes because a scheduled synthetic request, fired every five minutes, has been showing the same failure pattern since the rollout.

This lesson is the discipline of verifying that the trace context survives every hop, the synthetic request that catches a regression in seconds rather than in the next incident, and the failure modes that break the chain.

What it is

Trace propagation is the act of carrying the trace context across a service boundary. The context is the set of values that identify the current trace and span: trace_id, span_id, trace_flags, and the W3C-defined traceparent header that wraps them. The standard is the W3C Trace Context Recommendation; the OpenTelemetry SDK implements it as the default propagator.

The header format:

traceparent: 00-<trace-id>-<parent-id>-<flags>
example:    traceparent: 00-8f1d2c4e9a3b7f1d2c4e9a3b7f1d2c4e-c4e29b1a9b07ddee-01

The four fields:

• Version (00) — fixed for the W3C standard.
• Trace ID (8f1d...2c4e) — 16 bytes (32 hex chars), identifies the trace.
• Parent ID (c4e2...ddee) — 8 bytes (16 hex chars), identifies the current span.
• Flags (01) — 01 means the trace is sampled; 00 means the trace is not sampled. The SDK reads the flag and decides whether to record the trace.

A propagation test is a synthetic request that exercises every hop with a known traceparent and verifies, at each hop, that the request arrived with the same trace_id it was sent with. A passing test means the chain is intact. A failing test means one of the hops has dropped the header.

Why a sysadmin cares

Broken propagation is the most common reason a trace pipeline becomes useless. Three operational pains are specific to propagation regressions.

1. The trace tree that stops at the first hop. Every downstream service emits its own root span with a new trace_id. The trace view shows a forest of single-span traces, none of which has the dependency chain. The investigation is reduced to guessing which hop broke.
2. The metric that does not match the trace. The metric histogram says the system is slow; the trace view shows only the root service as slow. The discrepancy is because the downstream services are running in their own traces and their latency is hidden.
3. The exemplar that points at an empty trace. The dashboard shows a diamond icon; clicking it opens Tempo; the trace is a single span. The exemplar is correct; the trace it points at is incomplete because propagation was broken at the first hop after the metric was recorded.

How it works

The mental model is a header that travels with the request and is read at each hop to start the child span.

Hop 1: client → api-gateway
       sends: traceparent: 00-AAAA...BBBB-CCCC...DDDD-01
       api-gateway reads traceparent, starts a span with
         trace_id = AAA...BBBB
         parent_span_id = CCC...DDDD
         span_id = (new, server-generated)

Hop 2: api-gateway → checkout
       sends: traceparent: 00-AAAA...BBBB-(api-gateway span)-01
       checkout reads traceparent, starts a span with
         trace_id = AAA...BBBB  (same trace)
         parent_span_id = (api-gateway span)
         span_id = (new, server-generated)

Hop 3: checkout → payment-svc
       sends: traceparent: 00-AAAA...BBBB-(checkout span)-01
       payment-svc reads traceparent, starts a span with
         trace_id = AAA...BBBB  (same trace)
         parent_span_id = (checkout span)
         span_id = (new, server-generated)

At each hop, the receiver reads the traceparent header, extracts the trace_id, and uses it as the trace_id of the newly-created span. The parent_span_id becomes the parent of the new span. The trace tree grows by one level per hop, all sharing the same trace_id.

A propagation test exercises every hop and confirms, at each hop, that the trace_id is the one the test sent. The test must verify at the end of the chain, not the beginning — because the only signal that proves every hop preserved the header is that the leaf service has a span with the test’s trace_id.

The synthetic request

A propagation test is a request with a known trace_id that the test sends and then searches for in the trace backend.

# CONFIGURATION: a unique trace_id for this test run.
TRACE_ID="00000000000000000000000000000001"

# The traceparent header is constructed from the trace_id
# and a known parent_id.
PARENT_ID="0000000000000001"
TRACEPARENT="00-${TRACE_ID}-${PARENT_ID}-01"

# READ-ONLY: send the request with the traceparent header.
curl -X POST http://api-gateway.internal/checkout \
  -H "traceparent: ${TRACEPARENT}" \
  -H "Content-Type: application/json" \
  -d '{"order_id":"propagation-test"}'

{"status":"accepted","order_id":"propagation-test"}

The response confirms the request was received; it does not confirm propagation. The propagation confirmation is in the trace backend.

# READ-ONLY: query Tempo for the trace by ID.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  "https://tempo.internal.example.com/api/traces/${TRACE_ID}" \
  | jq '.resourceSpans | length'

1

A single trace resource exists. The trace contains one or more spans. The span count is the indicator of propagation depth:

# READ-ONLY: count the spans in the trace.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  "https://tempo.internal.example.com/api/traces/${TRACE_ID}" \
  | jq '[.resourceSpans[].scopeSpans[].spans[]] | length'

7

Seven spans means the chain reached at least seven spans. The expected number for the application’s call graph is the test’s pass condition; fewer spans means a hop dropped the header.

Verification at every hop

A coarse-grained test that only checks the leaf is enough to detect a regression in the chain. A fine-grained test that checks each hop individually is the operationally useful version: it pinpoints the broken hop.

The fine-grained test requires each hop to log the trace_id it received. The OTel SDK records the trace_id on every span, but the spans are only visible after they are exported. The propagation test logs the trace_id at the moment the header is read:

@app.route("/checkout", methods=["POST"])
def checkout():
    ctx = trace.get_current_span().get_span_context()
    logger.info(
        "propagation_check",
        extra={
            "trace_id": format(ctx.trace_id, "032x"),
            "span_id": format(ctx.span_id, "016x"),
            "service": "checkout",
        },
    )
    # ... actual work ...

The structured log line carries the trace_id. The propagation test queries Loki (or the structured logs backend) for the log line by trace_id and confirms the trace_id matches at every hop:

{="propagation_check"} |= "00000000000000000000000000000001"

The LogQL query returns the log line for the hop if and only if the trace_id was preserved. A missing log line is a broken hop.

Under the hood

How to configure it

A scheduled propagation test is the durable form of the verification. The test runs every minute, sends a request through the chain, and records the outcome.

A simple Kubernetes CronJob:

# propagation-test-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: trace-propagation-test
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: test
              image: registry.internal/propagation-test:latest
              env:
                - name: TRACE_ID
                  value: "00000000000000000000000000000001"
                - name: PARENT_ID
                  value: "0000000000000001"
                - name: ENTRY_URL
                  value: "http://api-gateway.internal/checkout"
                - name: TEMPO_URL
                  value: "https://tempo.internal.example.com"
                - name: EXPECTED_SPANS
                  value: "7"
                - name: TEMPO_USER
                  valueFrom:
                    secretKeyRef:
                      name: tempo-credentials
                      key: username
                - name: TEMPO_PASS
                  valueFrom:
                    secretKeyRef:
                      name: tempo-credentials
                      key: password
              command: ["/usr/local/bin/propagation-test"]
          restartPolicy: OnFailure

The container image runs the test. The test:

1. Sends the request with the W3C traceparent header.
2. Waits five seconds for the spans to flush.
3. Queries Tempo for the trace by ID.
4. Counts the spans in the trace.
5. Compares the count to the expected count.
6. Records the outcome to a metric or log.

The script:

#!/usr/bin/env bash
set -euo pipefail

TRACE_ID="${TRACE_ID}"
PARENT_ID="${PARENT_ID}"
TRACEPARENT="00-${TRACE_ID}-${PARENT_ID}-01"
ENTRY_URL="${ENTRY_URL}"
TEMPO_URL="${TEMPO_URL}"
EXPECTED_SPANS="${EXPECTED_SPANS}"

curl -fsS -X POST "${ENTRY_URL}" \
  -H "traceparent: ${TRACEPARENT}" \
  -H "Content-Type: application/json" \
  -d '{"order_id":"propagation-test"}' >/dev/null

sleep 5

ACTUAL_SPANS=$(curl -fsS -u "${TEMPO_USER}:${TEMPO_PASS}" \
  "${TEMPO_URL}/api/traces/${TRACE_ID}" \
  | jq '[.resourceSpans[].scopeSpans[].spans[]] | length')

if [[ "${ACTUAL_SPANS}" -ge "${EXPECTED_SPANS}" ]]; then
  echo "PASS: trace_id=${TRACE_ID} spans=${ACTUAL_SPANS}"
  exit 0
else
  echo "FAIL: trace_id=${TRACE_ID} expected=${EXPECTED_SPANS} actual=${ACTUAL_SPANS}"
  exit 1
fi

A failure exits non-zero; the CronJob records the failure in its job status; Prometheus or the job-runner records the failure count as a metric; the alerting rule fires on the metric.

How to validate it

Three checks confirm the test is wired correctly.

# READ-ONLY: confirm the test job has run recently.
kubectl get jobs -l job-name=trace-propagation-test \
  -o jsonpath='{.items[*].status.completionTime}' \
  | tr ' ' '\n' | tail -5

2026-08-13T03:14:00Z
2026-08-13T03:15:00Z
2026-08-13T03:16:00Z
2026-08-13T03:17:00Z
2026-08-13T03:18:00Z

Recent timestamps mean the job is firing on schedule. The next check is the outcome:

# READ-ONLY: confirm the test reported a pass.
kubectl logs -l job-name=trace-propagation-test --tail=3 \
  | grep -E '^(PASS|FAIL)'

PASS: trace_id=00000000000000000000000000000001 spans=7
PASS: trace_id=00000000000000000000000000000001 spans=7
PASS: trace_id=00000000000000000000000000000001 spans=7

A sequence of PASS lines means the chain is intact and the expected span count is being met. The third check is the trace shape:

# READ-ONLY: confirm the trace has the expected service set.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  "https://tempo.internal.example.com/api/traces/${TRACE_ID}" \
  | jq '[.resourceSpans[].resource.attributes[]
        | select(.key == "service.name") | .value.stringValue] | unique'

[
  "api-gateway",
  "checkout",
  "payment-svc",
  "postgres-proxy"
]

The unique set of service.name values matches the expected hop set. A hop whose service is absent from the trace is a hop that dropped the header.

How it can fail

Five failure modes specific to propagation.

1. The reverse proxy that strips unknown headers. A new nginx or Envoy configuration allow-lists a set of request headers and silently drops everything else. The W3C traceparent is not on the allow-list. Symptom: the propagation test reports spans=1 immediately after the proxy is reconfigured. Cause: the proxy was reconfigured without the team knowing. The fix is to add traceparent and tracestate to the allow-list at every ingress and egress point.
2. The service mesh that rewrites headers. A service mesh sidecar strips or rewrites the traceparent header because of a custom header policy. Symptom: the propagation test reports spans=2 after the mesh upgrade. Cause: the mesh version introduced a header policy change. The fix is to add the W3C header names to the mesh’s allow-list.
3. The authentication layer that strips headers. A custom auth proxy logs and strips every header before forwarding. Symptom: the propagation test reports spans=1 immediately after the auth proxy is deployed. Cause: the auth proxy was not aware of the W3C header. The fix is to allow-list traceparent in the auth proxy’s header policy.
4. The async boundary that does not propagate. A service enqueues work onto a queue and returns immediately; a consumer picks up the work and processes it. The queue message does not carry the trace context. Symptom: the consumer’s span has a different trace_id than the producer’s. Cause: the message producer did not inject the trace context into the message headers. The fix is to inject the context at the producer and extract it at the consumer; the OTel SDK supports both for the standard messaging libraries.
5. The SDK that does not use the W3C propagator. A service initialises the SDK with the B3 propagator instead of the W3C propagator. Symptom: the traceparent header from the client is ignored; the service emits its own x-b3-traceid header. Cause: the SDK was configured with the legacy propagator from a Jaeger migration. The fix is to set the propagator to the W3C TraceContext propagator.

How to troubleshoot it

When the propagation test fails, the order matters.

1. Inspect the entry hop. Use a known traceparent and inspect the traceparent header at the first hop. If the header is missing, the client or the network between the client and the service stripped it.
2. Inspect each proxy. Run curl -v to see the headers received at each hop. A proxy that is supposed to forward traceparent and does not is the broken hop.
3. Inspect the SDK propagator. OTEL_PROPAGATORS should be set to tracecontext (the W3C propagator). A value of b3 or jaeger indicates a legacy propagator; the SDK will not recognise the W3C traceparent header.
4. Inspect the service mesh policy. Istio’s meshConfig.outboundListenerPolicy and Linkerd’s proxy.inbound.headersPolicy have explicit header allow-lists. The W3C header names must be present.
5. Inspect the SDK initialisation order. The propagator must be set before the first span is created. A propagator set after the first inbound request has been processed is too late; that request will not have a context.

Security implications

Propagation is a header-passing mechanism. The header carries a trace_id and a span_id; both are random and carry no sensitive data. The propagation test does not introduce new attack surface.

Three operational rules:

• Treat the tracestate header as semi-sensitive. The W3C tracestate header is an open key-value pair that vendors can use to carry vendor-specific context. A malicious actor could craft a header that attempts to inject context; the SDK should sanitise the header before trusting it. The default W3C propagator does this.
• Do not allow the traceparent header to influence routing. A header that says “this is the traceparent” should not be used as a routing key. The propagation mechanism is orthogonal to the request routing.
• Audit the propagator configuration. A service that uses the B3 propagator is a legacy from a Jaeger migration; the audit should confirm the W3C propagator is in use.

Performance implications

The propagation test is a single request per minute. The cost on the chain is one request’s worth of work; the cost on the test infrastructure is one HTTP call and one Tempo lookup. The cost is negligible.

The cost of a propagation regression is the lost investigation time during incidents. The cost-benefit calculation is straightforward.

Production guidance

• Run the propagation test on a schedule, not on demand. The test must catch regressions that happen during deploys or configuration changes when no engineer is paying attention.
• Pin the expected span count. The application’s call graph is known; the expected number of spans per request is known. The test should fail when the count drops below the expected number, regardless of the cause.
• Run the test from the same network as production traffic. A test that runs from a separate network may exercise different proxies and load balancers than production traffic; the test would miss a production-only regression.
• Allow-list the W3C headers at every ingress and egress. The header names are traceparent and tracestate. The allow-list must be present at every proxy, mesh sidecar, and authentication layer.

Verification

You should now be able to answer:

• What are the four fields of the W3C traceparent header?
• Why does a propagation test verify the trace_id at the leaf service rather than at the entry service?
• What is the most common operational cause of broken propagation?
• Why is a scheduled propagation test better than an on-demand verification?
• What two headers must be allow-listed at every proxy in the path?

Quiz