Trace ID and Span ID

A trace_id arrives in a Slack message from a customer success manager. The on-call engineer pastes it into the Grafana trace explorer. Tempo returns the full trace: nine spans across four services, 420 ms critical path, one span in error. The same trace_id, queried later by another engineer, returns the same answer. That is the property the identifier design buys you: one identifier, one trace, look it up directly. No schema, no join, no index lookup that misses.

What it is

OpenTelemetry defines two identifiers on every span:

• trace_id — 16 bytes (128 bits), lowercase hexadecimal, 32 characters. Identifies the trace. Every span in one trace shares the same trace_id. Generated once at the root span and copied into every child.
• span_id — 8 bytes (64 bits), lowercase hexadecimal, 16 characters. Identifies a single span within a trace. Unique within the trace.

Both identifiers are opaque; they carry no inherent meaning. They are not UUIDs in the textual sense — they are random byte sequences — but the textual representation matches the canonical UUID representation in size and case.

A third identifier — parent_span_id — is an 8-byte span_id that names the parent. The root span has a parent_span_id of 16 zero hex characters (0000000000000000); this is the conventional “no parent” marker.

W3C Trace Context additionally constrains the textual form: traceparent: 00-<32 hex>-<16 hex>-<flags>. Version 00 mandates 16-byte trace_id and 8-byte span_id. Future versions may extend this; SDKs must accept any size the standard defines and emit 16 / 8 today.

Why a sysadmin cares

The identifiers are not decoration. Three operational properties depend on them being right:

• Lookup by ID. The trace UI’s “open trace by ID” feature calls /api/traces/{trace_id} on Tempo. A malformed or truncated identifier returns 400 with no diagnostic. The engineer’s first ten minutes are spent figuring out whether the trace exists.
• Correlation across signals. The trace_id appears in application logs, in Loki-derived metrics, in Prometheus exemplars, in error trackers, in customer support tickets. A 16-byte identifier is the contract that lets a single identifier move from one system to another.
• Storage layout. Tempo’s trace blocks are named with the trace_id. The compactness of the hex representation (4bf92f3577b34da6a3ce929d0e0e4736 is 32 characters) is what makes the object-store path short and the per-trace-object size manageable.

A trace_id collision (two different traces with the same id) would silently merge two unrelated traces into one block in Tempo. With 128 bits of randomness, the chance of collision is negligible in practice; the operational risk is poor identifier generation, not the math.

How it works

A typical request flow looks like this, with the identifiers visible at every boundary:

client                gateway              checkout             payment
   |                    |                    |                    |
   | -- traceparent: 00-aaaa....-bbbb....-01 --> |                    |
   |                    |                    |                    |
   |                    | span created:      |                    |
   |                    |   trace_id=aaaa... |                    |
   |                    |   span_id =cccc.... |                    |
   |                    |   parent   =bbbb... |                    |
   |                    |                    |                    |
   |                    | -- traceparent: 00-aaaa....-cccc....-01 --> |
   |                    |                    |                    |
   |                    |                    | span created:      |
   |                    |                    |   trace_id=aaaa... |
   |                    |                    |   span_id =dddd.... |
   |                    |                    |   parent   =cccc... |
   |                    |                    |                    |
   | <---- 200 OK ----- | <------ 200 OK --- | <-- 200 OK ------ |

Four observations:

• trace_id is constant end to end.
• span_id is freshly minted by every service that creates a span.
• parent_span_id of the child matches the span_id of the immediate caller.
• The root span (span_id = bbbb... in the gateway) has parent_span_id = 0000000000000000 (the all-zero marker).

Tempo’s /api/traces/{trace_id} endpoint accepts the 32-char hex form; the response is the full trace block. The same trace_id works in any TraceQL filter as an intrinsic field: { trace_id = "4bf92f3577b34da6a3ce929d0e0e4736" }.

Under the hood

How to configure it

OpenTelemetry SDKs generate identifiers internally; there is nothing to configure at the SDK level beyond selecting a propagator that emits W3C-compliant traceparent headers. Configure the SDK to use the standard identifier lengths and to log the trace_id at the access-log layer:

# app.py -- Python logging filter that adds trace_id to every log line
import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        if span and span.get_span_context().is_valid:
            record.trace_id = span.get_span_context().trace_id
            record.span_id = span.get_span_context().span_id
        else:
            record.trace_id = None
            record.span_id = None
        return True

handler = logging.StreamHandler()
handler.addFilter(TraceIdFilter())
logging.getLogger().addHandler(handler)

The access log line then looks like:

2026-08-13T14:22:11Z POST /checkout 200 320ms trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7

That trace_id value is what the on-call engineer pastes into Grafana.

Tempo side — the default identifier length is fixed; there is no configuration to change. What you do configure is the storage path layout, which uses the first bytes of the trace_id as a prefix:

# /etc/tempo/tempo.yaml
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-blocks
      endpoint: s3.internal:9000
    wal:
      path: /var/tempo/wal

# Optional: configure the search index to drop trace_id from the indexed
# columns (it is the key, not a value).
search:
  max_duration: 0   # 0 = no retention cap; tune for storage budget

How to validate it

# 1. Generate a request and capture the trace_id from the access log.
TRACE=$(grep -o 'trace_id=[0-9a-f]\{32\}' /var/log/checkout/access.log | tail -1 | cut -d= -f2)
echo "$TRACE"

# 2. The ID must be exactly 32 lowercase hex characters.
echo -n "$TRACE" | wc -c          # 32
echo "$TRACE" | grep -E '^[0-9a-f]{32}$' && echo OK || echo BAD

# 3. Tempo returns the trace block for this ID.
curl -s -o /tmp/trace.json -w '%{http_code}\n' -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE"
# Expected: 200

jq '.batches[].scopeSpans[].spans[] | {trace: .traceId, span: .spanId, parent: .parentSpanId}' /tmp/trace.json

# Expected (illustrative):
# { "trace": "4bf92f3577b34da6a3ce929d0e0e4736",
#   "span":  "00f067aa0ba902b7",
#   "parent": "0000000000000000" }
# { "trace": "4bf92f3577b34da6a3ce929d0e0e4736",
#   "span":  "1a2b3c4d5e6f7081",
#   "parent": "00f067aa0ba902b7" }
# { "trace": "4bf92f3577b34da6a3ce929d0e0e4736",
#   "span":  "9b8c7d6e5f4a3210",
#   "parent": "1a2b3c4d5e6f7081" }

# 4. Every span must share the same trace_id. Any span with a
# different trace_id is an orphan (covered in lesson 01).
jq -r '.batches[].scopeSpans[].spans[].traceId' /tmp/trace.json | sort -u | wc -l
# Expected: 1

If step 3 returns 404, the trace has not reached Tempo yet (the batch processor flushes every 5 s by default). If it returns 400, the identifier is malformed — check the length and character set. If it returns 200 but step 4 returns more than one distinct trace_id, the trace has been split across two storage blocks and one is corrupt.

How it can fail

1. Malformed identifier on input. A custom propagator emits traceparent: 00-aaaa...-bbbb...-01 where the trace_id is 16 hex chars instead of 32. W3C mandates 32 hex chars for version 00. The receiver treats the header as invalid, drops the parent context, and starts a new trace. Symptom: the trace exists but every span after the malformed hop has a new trace_id.
2. Uppercase hex. A non-compliant producer emits AAAA.... instead of aaaa..... W3C requires lowercase. Some receivers accept both; others silently drop. Symptom: inconsistent propagation that works for one consumer and not another.
3. All-zero trace_id. A library bug or a misuse of the API sets trace_id = 0. OpenTelemetry treats the SpanContext as invalid and the span is dropped. Symptom: the application’s span is missing entirely from Tempo.
4. Collapsing distinct traces into one block. A bug causes two unrelated requests to share a trace_id. Tempo stores them as one block. Symptom: the trace UI shows interleaved spans from different users under one trace.
5. trace_id not in the access log. The application forgets to log the trace context. Symptom: the on-call engineer has a customer-supplied request ID but cannot convert it to a trace_id; the investigation requires searching by attribute instead of by direct ID.
6. Trace ID logged but truncated. The log format uses a fixed-width field and the last 16 hex chars are dropped. Symptom: every “trace_id” in the logs is malformed; no lookup succeeds.

How to troubleshoot it

Security implications

Trace and span identifiers are random 128 / 64-bit values. They are not secrets and are safe to log, return in response headers, and share across trust boundaries. They do, however, enable correlation: a trace_id that appears in a customer support ticket and again in an application log is the pivot that turns a vague complaint into a precise investigation. Treat them with the same discipline as any other correlation ID: log them, do not echo them back to the client unless the API design explicitly says so, and never use them as authorisation tokens (a known attacker tactic is to guess recent identifiers and replay them against endpoints that trust them).

Performance implications

The identifier sizes are fixed and small. The performance implication of identifier design is not on the wire (32 / 16 hex chars is trivial) but in the index: every distinct trace_id is a row in the trace block store, and every distinct trace_id for which there is a search-index sample is a row in the index. This is the operational reason sampling exists: not all traces need to land in storage, and the ones that do not are simply not created. Sampling is the dial that controls the cardinality of trace_id over time.

Production guidance

• Standardise the log format: trace_id=<32 hex> span_id=<16 hex>.
• Validate identifiers at ingress. A W3C-compliant propagator drops invalid traceparent headers silently; an explicit validator at the edge surfaces the malformed-header failure mode early.
• Treat the trace_id as a primary key in any cross-system correlation — between Loki, Tempo, error trackers, and the change log.

Verification

You should now be able to answer:

• How many bytes are in an OpenTelemetry trace_id, and how is it rendered in text?
• What marker indicates a root span’s parent_span_id?
• How does Tempo derive an object’s path from a trace_id?
• What HTTP status does Tempo return for a malformed identifier?
• Why is the trace search index not the trace itself?

Quiz