Trace Anatomy

Checkout fails for one in three attempts at 03:00. The error rate panel is red, the dependency-latency panel is flat, and the change log is empty. The on-call engineer opens Grafana, picks Explore > TraceQL, types \{ service.name = "checkout" && status = error \}, and within a minute points to the offending span. The metric told them something was wrong; the trace tells them where. That is what trace anatomy buys you.

What it is

A trace is the journey of a single logical request through a distributed system, encoded as a directed acyclic graph of spans that share one identifier. A span is a named, timed unit of work: it has a start time, an end time, a set of attributes, a status, optional events, and links to other spans. One span has no parent and is the root span; every other span is the child of exactly one parent. The parent / child links form the trace tree. The root span is the entry point — typically the incoming HTTP request or the head of a queue consumer.

This is the canonical OpenTelemetry data model and the storage unit in Grafana Tempo. Every span carries the same trace_id. Together they form one trace.

Why a sysadmin cares

Metrics answer “how much / how often”. Logs answer “what happened”. Traces answer “where did the time go and which dependency is on fire”. The three are not interchangeable:

• A trace is the only signal that links a user-visible latency spike to the specific dependency that caused it.
• A trace is the only signal that proves a deployment did not regress a sub-request that no metric panel covers.
• A trace is the only signal that survives a request moving across async, queue, or event boundaries without breaking the causal chain — provided context was propagated correctly.

Without traces, “checkout is slow” becomes detective work in metric panels and grepped logs. With traces, the question becomes “which span took how long”.

How it works

Picture a single POST /checkout request landing on the checkout service, fanning out to four downstream services, two of which issue database calls. The trace looks like this:

trace_id=4bf92f3577b34da6a3ce929d0e0e4736

checkout POST /checkout                         320 ms  root
|-- auth verify_token                            18 ms
|-- cart get_cart                                22 ms
|   `-- db SELECT FROM carts WHERE id=42          8 ms
|-- payment charge                              240 ms   <- critical path
|   |-- fraud check_risk                         35 ms
|   `-- db INSERT INTO payments                  18 ms
|-- shipping calc_rate                           28 ms
|   `-- http GET shipping-api                    21 ms
`-- receipt render                               10 ms

Properties of this picture that matter operationally:

• Every span shares the same 16-byte trace_id.
• The root span has no parent. Every other span names its parent with an 8-byte parent_span_id.
• Durations nest. The root span (320 ms) is the wall-clock time from the first byte received to the last byte sent. Children account for the time spent inside each unit of work; sibling spans run in parallel.
• The critical path is the longest chain of sequentially-dependent children (checkout > payment > fraud > db). Optimising anything off the critical path does not reduce end-to-end latency.
• Tree shape encodes concurrency. Two sibling spans inside one parent ran in parallel from the parent’s perspective.

Under the hood

How to configure it

The fastest path to a working trace is OpenTelemetry auto- instrumentation on the application plus the OpenTelemetry Collector as a gateway. The Collector fans spans out to Tempo.

Application side — Python (Flask) with the OTel SDK and auto-instrumentation:

# app.py  -- enable OTel tracing
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "checkout",          # mandatory -- search key in Tempo
    "service.namespace": "shop",
    "service.version": "1.42.0",
    "deployment.environment": "prod",
})

provider = TracerProvider(resource=resource)
# OTLP gRPC to the Collector on the standard port 4317
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True))
)
trace.set_tracer_provider(provider)

Collector side — OTLP receiver, batch processor, Tempo exporter:

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:               # the application talks gRPC on 4317
        endpoint: 0.0.0.0:4317
      http:               # and HTTP/protobuf on 4318 (useful for curl tests)
        endpoint: 0.0.0.0:4318

processors:
  batch:                 # mandatory for production -- avoid one-span-per-RPC
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Grafana Alloy side — same effect, different syntax:

# /etc/alloy/config.alloy
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output { traces = [otelcol.processor.batch.default.input] }
}

otelcol.processor.batch "default" {
  timeout = "5s"
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

otelcol.exporter.otlp "tempo" {
  client { endpoint = "tempo:4317" }
}

The application SDK is responsible for emitting the root span (usually done by an auto-instrumentation library hooking the HTTP framework) and for propagating the trace context to outgoing calls. The Collector is responsible for transport, batching, and fan-out to backends. Tempo is responsible for storage and query.

How to validate it

Send a request that exercises the full chain, then query Tempo twice: once by TraceQL to find the trace, and once by ID to read its full structure.

# 1. Generate a request. The response includes the trace ID.
TRACE=$(curl -s -X POST http://shop.internal/checkout \
  -H 'Content-Type: application/json' \
  -d '{"cart_id":42}' | jq -r .trace_id)

# 2. Read the full trace by ID.
curl -s -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" | jq '.batches[].scopeSpans[].spans[] | {name, duration_ns: (.endTimeUnixNano - .startTimeUnixNano), parent: .parentSpanId}'

# 3. Find the slowest traces for the service in the last 15 minutes.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ service.name = "checkout" && duration > 1s }' \
  --data-urlencode 'limit=20' \
  --data-urlencode 'since=15m' \
  http://tempo.internal:3200/api/search | jq '.traces[].traceID'

Expected output for step 2:

{ "name": "POST /checkout",   "duration_ns": 320041000, "parent": "0000000000000000" }
{ "name": "verify_token",     "duration_ns":  18022000, "parent": "0a1b2c3d4e5f6071" }
{ "name": "charge",           "duration_ns": 240110000, "parent": "0a1b2c3d4e5f6071" }
{ "name": "check_risk",       "duration_ns":  35080000, "parent": "1a2b3c4d5e6f7081" }
{ "name": "INSERT payments",  "duration_ns":  18041000, "parent": "1a2b3c4d5e6f7081" }

The zero parent value is the root span. The 320 ms parent and the sum of child durations should reconcile — if the root is shorter than the sum of its children, the SDK is recording clock times incorrectly.

How it can fail

1. Orphaned spans. A child span arrived without its parent because the parent’s export batch flushed later. Tempo displays the orphan at depth zero as if it were a root, and the real root appears as a separate, unrelated trace. Symptom: the trace you search for has half the spans you expect, and a second trace with the same service and timestamp appears nearby.
2. Sampling-induced gaps. The application uses tail-based sampling with a misconfigured policy; the parent was kept but a child was dropped. The trace renders but the critical path stops mid-flight. Symptom: the slowest span is the leaf and the parent looks suspiciously short.
3. Lost context at the edge. An API gateway or a load balancer strips the traceparent header on rewrite. The upstream service starts a new trace. Symptom: every trace ends at the edge service; the backend shows zero traces originating from the gateway.
4. Clock skew. Host clocks differ by tens of milliseconds. Child spans appear to start before their parent. Symptom: flame graph shows children outside the parent’s bar; durations are negative in the JSON view.
5. Collector in the loop, app not exporting to it. The application still pushes spans to the old collector address; the new one is silent. Symptom: Tempo sees a single span per service with service.name set to the host instead of the logical service name; the search index has gaps.
6. Wrong protocol port. The application exports OTLP/HTTP to the collector’s gRPC port. Symptom: connection refused, one-line error in the application log, but no trace and no retry storm — the failure is silent on the collector side.

How to troubleshoot it

Security implications

Trace IDs are not secrets — they are 128-bit random values — but they can leak request metadata. A trace ID correlated with an exposed log line or response header can let an outsider pivot to all logs from that request. PII in span attributes (names, emails, JWTs) is a much larger exposure: spans are retained for days or weeks, and the search index exposes them to anyone with Tempo read access. Treat span attributes with the same discipline as application logs: redact at the SDK before export, never rely on a backend filter.

Performance implications

A single trace with 50 spans and 30 attributes per span is roughly 20-40 KB on the wire. At 1000 requests per second with 5% sampling, that is 1-2 MB/s to the collector, 50-100 KB/s to Tempo. The collector’s batch processor smooths this out; the load is steady rather than spiky. The expensive part is storage: Tempo retains the trace block, and a full-fidelity 14-day retention of one million traces per day is a non-trivial disk budget. Tune compactor.block_retention and the trace-search index size in proportion to traffic, not to disk available.

Production guidance

• Keep the application SDK simple. One TracerProvider, one OTLP exporter, one batch processor. Per-request or per-thread configuration is a smell.
• Use head-based sampling at the edge (gateway, ingress) so the sampling decision is consistent across services. Tail-based sampling is for narrowing errors and slow traces after the fact, not for first-pass collection.
• Set service.name once, at SDK initialisation, and never override it from request context. A trace search index with 2000 distinct service.name values is useless.

Verification

You should now be able to answer:

• What is a trace, structurally, in OpenTelemetry’s data model?
• What identifies the root span, and what identifies a child?
• What is the critical path, and how do you read it in a flame graph?
• What does Tempo use as the storage key for a trace?
• How do you tell, from a single trace in the UI, whether the collector dropped a span versus the application not emitting it?

Quiz