Questions Only Traces Answer

The dashboard says checkout latency p99 is 1.4 s. The SLO budget is burning. The on-call engineer needs to know which call is responsible before they can fix anything. The metric tells them the system is slow. The metric does not tell them which call is slow. Only a trace answers that question without an extra deploy.

This lesson is the catalogue of questions that traces answer cleanly and the matching ones that metrics and logs answer only with great effort. The list is the design test for instrumentation: if you cannot name the question a piece of telemetry answers, the telemetry is decoration.

What it is

A distributed trace is the per-request record of a single operation across every service it touches. Each hop in the request is one span; spans are linked by a shared trace ID; each span carries a name, a service, a start time, a duration, a status, and a set of attributes. A complete trace answers four operational questions that no aggregate signal can answer on its own:

  Four questions only traces answer
  --------------------------------
  1. Where did the latency go?
     Which span inside the request took the most wall-clock time?

  2. Which dependency failed?
     Which remote call returned the error, and which
     downstream calls did that error propagate to?

  3. Where did the request die?
     In a partial trace, which span is the leaf that never
     closed, and what state was the parent in when it gave up?

  4. What is the chain of cause?
     Which span triggered which other span, in what order,
     and with what correlation IDs at each hop?

Each question pivots on a span attribute, a span relationship, or a span timing that an aggregate signal does not preserve.

Why a sysadmin cares

Metrics answer questions about populations; traces answer questions about individuals. The population view is what dashboards and alerts need. The individual view is what incident response needs. A team with only metrics knows the system is in trouble; a team with traces knows which request to open.

Three failure shapes appear when traces are missing:

1. The slow dependency no one can name. A histogram shows latency is elevated. The histograms for each dependency look the same. The engineer cannot pick one to investigate first and ends up checking all of them. The investigation takes twenty minutes instead of three.
2. The error 503 that no log explains. Application logs show a generic upstream error. The upstream service has no log of the request, because the request never arrived — it timed out at the load balancer. Without a trace, the engineer has to instrument the path between the load balancer and the application to find the missing hop.
3. The intermittent failure that no alert catches. A request fails one in a thousand times. The aggregate error rate is below the alert threshold. The trace of any single failed request shows the cause clearly; without traces, no record of the failure exists at all.

How it works

The mental model is a tree. Each trace is one tree; each node is one span; each edge is a parent-child relationship.

Trace: 8f1d...a3c  (HTTP POST /checkout, 1.42 s)
|
+-- span api-gateway.checkout           1.42 s   (root)
      |
      +-- span checkout.cart.read        0.012 s  (DB)
      +-- span checkout.pricing.lookup   0.083 s  (HTTP)
      |     |
      |     +-- span pricing-svc.calc    0.078 s  (in-process)
      |
      +-- span checkout.payment.charge   1.31 s   (HTTP)  <-- slow
      |     |
      |     +-- span payment-svc.charge  1.30 s   (HTTP)
      |     |     |
      |     |     +-- span pg.charge.tx  1.29 s   (DB)    <-- slowest
      |
      +-- span checkout.receipt.write    0.014 s  (DB)

The leaf spans are the work units. The non-leaf spans are the container units. A span’s own duration includes the time its children spent waiting; a span’s exclusive time is the rest. For the root span above, the exclusive time is 1.42 - (0.012 + 0.083 + 1.31 + 0.014) = 0.001 s; almost everything was spent inside children.

The four questions map to four operations on the tree:

• Where did the latency go? Sort spans by duration; the longest synchronous chain from root to leaf is the critical path.
• Which dependency failed? Find the span with status=error; its parent shows the caller; its children (if any) show the propagation.
• Where did the request die? Find the leaf span that never closed. The parent either timed out or recorded an error pointing at the leaf.
• What is the chain of cause? Walk the parent links from the failing span upward; each parent shows a level of context.

The data model

Each span is a small structured record. The fields are standardised by the OpenTelemetry semantic conventions so that Tempo, Jaeger and other backends can render traces from any instrumented language the same way:

trace_id        8f1d...a3c    (32 hex chars; shared by every span)
span_id         c4e2...9b1    (16 hex chars; unique per span)
parent_span_id  9a07...2dd    (16 hex chars; absent for the root)
name            "pg.charge.tx"
service.name    "checkout"
duration        1.29s
start_time      2026-08-13T03:14:22.184Z
status.code     ERROR
status.message  "deadline exceeded after 1.3s"
attributes:
  db.system            postgresql
  db.statement         INSERT INTO charges ...
  net.peer.name        pg-primary.internal
  http.response.status_code 504
events:
  - name "exception"
    time 2026-08-13T03:14:23.491Z
    attributes { exception.type "TimeoutError" }

The trace_id is the join key. Logs that carry the same trace_id join onto the trace; metrics that carry exemplars (see lesson 03) point back at a sample trace; an alert on elevated error rate can include a link to the most recent failing trace.

Under the hood

How to configure it

Instrumentation in the application is the right place to start. For a Python service using the OpenTelemetry SDK, a span for an outbound dependency is one decorator away:

from opentelemetry.instrumentation.requests import RequestsInstrumentor

RequestsInstrumentor().instrument()

That single call wraps every requests call in a span with the standard http.* attributes. The collector at the edge is the OpenTelemetry Collector or Grafana Alloy, configured to receive OTLP and forward to Tempo:

# /etc/alloy/config.alloy
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.internal.example.com:4317"
    tls {
      ca_file = "/etc/ssl/certs/ca-certificates.crt"
    }
  }
}

The service.name resource attribute is the pivot for every later query. Set it once, in the application’s resource builder, and every span the service emits inherits it. Hard-coding it in the collector works only if every service has a single identity; in a polyglot fleet, the application owns the attribute.

How to validate it

Three checks confirm the trace pipeline is live.

# READ-ONLY: confirm spans are arriving at Tempo.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  https://tempo.internal.example.com/api/search?limit=5 \
  | jq '.traces | length'

5

A non-zero count means recent traces exist. The span count and the service names are the next check:

# READ-ONLY: confirm the trace has spans from the expected services.
tempo-cli trace show 8f1d...a3c 2>&1 \
  | jq '.resourceSpans[].scopeSpans[].spans[] | .name' \
  | sort -u

"api-gateway.checkout"
"checkout.cart.read"
"checkout.pricing.lookup"
"checkout.payment.charge"
"checkout.receipt.write"
"pg.charge.tx"

The expected span names appear once. If the list contains only the root span and no children, propagation is broken at the first hop; see lesson 05.

A TraceQL query from the Grafana Explore UI confirms the same result via the user-facing surface:

{ resource.service.name = "checkout" && span.http.status_code = 504 }

That returns the recent traces whose checkout path produced a 504. Open the slowest one. The trace view shows the leaf span that produced the timeout and the parent chain that waited on it.

How it can fail

Six failure modes that turn traces from an investigation tool into a misleading artifact.

1. The trace with a single span. The root span exists, the children do not. Symptom: tempo-cli trace show returns one span with no children. Cause: propagation context is missing at the first outbound hop. The instrumentation wraps the outbound call but does not propagate the W3C traceparent header. Every downstream call becomes a new trace.
2. The trace with the wrong service name. Every span in the trace carries service.name = "unknown_service" or the default SDK placeholder. Symptom: TraceQL queries filter by resource.service.name and return empty. Cause: the resource attribute is set inside a child process that exits before the spans are flushed, or the SDK is initialised after the first span is emitted.
3. The trace with no attributes. The spans arrive but every attributes map is empty. Symptom: queries by http.url, db.statement, or any other domain attribute return nothing. Cause: the SDK is configured with a default attributes_limit of zero, or a span processor is dropping the attribute map on a size limit.
4. The trace that arrives three hours late. The trace is correct; it just sits in the SDK buffer for hours. Symptom: Tempo queries for the last fifteen minutes are empty; the trace appears later. Cause: the batch span processor is configured with a very large max_queue_size and schedule_delay, or the OTLP endpoint is unreachable and the retry is silent.
5. The trace with the right name and the wrong shape. A critical section is missing its span. Symptom: a known dependency is absent from the trace tree, and the root span’s exclusive time is implausibly high. Cause: a section of the code was instrumented manually and the manual span was closed before its child operations finished.
6. The trace whose status.code is set to OK on an error. The SDK auto-sets status.code = ERROR only when an exception reaches the span. Symptom: queries for failing traces return nothing because the status was overwritten to OK by a finally block. Cause: the developer called span.set_status(OK) to clear an earlier status, and the exception was swallowed before it reached the SDK.

How to troubleshoot it

When the trace pipeline is up but the answer is missing, the order matters.

1. Inspect the SDK exporter log. A warning that says ExportFailed with a Retryable flag tells you the OTLP endpoint is unreachable or the auth credentials were rejected. Fix the endpoint before reading any further signal.
2. Inspect the collector’s receiver metric. otelcol_receiver_accepted_spans should advance at the rate your service emits spans. A counter that is not advancing means the receiver is up but no traffic is reaching it.
3. Confirm the resource attributes on a single span. Run tempo-cli trace show on a recent trace and inspect the first span’s resource.attributes. The service.name and service.version should both be set. An empty service.name is the most common cause of “the trace is there but my filter returns nothing”.
4. Confirm the span tree shape. The number of spans per trace should match the depth of your call graph. A trace with one span has lost propagation; a trace with one span and a deep tree shape has lost child instrumentation.
5. Search by status, then by name. TraceQL { status = error } returns every error span in the index for the time window. Walk the parents from there. The narrowest filter is the last step, not the first.

Security implications

Traces carry the same data the application handles. Headers, query parameters, and SQL statements that ride in span attributes will end up in the trace store. Three production rules:

• Strip high-cardinality identifiers before they reach the attribute map. A request ID or a session cookie should not be a span attribute; it would make every trace unique and unsampleable.
• Treat the trace backend as sensitive. Tempo stores spans in object storage. The bucket must be encrypted at rest, the read API must require authentication, and the retention must match the data-protection regime for the data inside the trace.
• Redact the same fields in spans that you redact in logs. Email addresses, authorisation headers, and tokens that are filtered in the log pipeline must be filtered in the SDK before the span is exported. The default OTel HTTP instrumentation sets http.request.header.authorization from the request headers; most teams want to disable that attribute or scrub it before export.

Performance implications

The SDK adds CPU and memory cost on the application host. The order of magnitude for a service instrumented with the default SDK is single-digit percent of CPU and tens of MiB of heap, per host. The expensive patterns are:

• Per-request attribute allocation. A map[string]string with hundreds of entries per span turns into many short-lived allocations. Use the SDK’s attribute builders; they reuse memory.
• Sampling at the tail, not at the head. Sampling at the collector forces every application to emit 100% of spans and then discard most at the edge. Sampling at the head (a probability decision inside the SDK) reduces SDK cost as well as backend cost.
• High-cardinality attribute values. A span with a unique user_id attribute creates a unique trace. The Tempo block size grows; the index grows; the query cost grows. Cap the cardinality per attribute; lesson 06 covers this in detail.

Production guidance

• Design around the question, not the tool. For each service, write down the four questions from this lesson and confirm the instrumentation answers each one. A service that has spans but cannot answer “which dependency failed” is over-instrumented for coverage and under-instrumented for the question.
• Set service.name and service.version as resource attributes. They are the join keys for every later filter. The application owns them; the collector does not.
• Standardise span names. Use the OTel semantic conventions for HTTP servers (HTTP {method} {route}), database clients ({db.operation} {target}), and messaging clients ({destination} {operation}). A free-form span name is impossible to aggregate in TraceQL.
• Keep the SDK batch processor at its defaults. The defaults are tuned for backend ingest; tune them only with measured reason.

Verification

You should now be able to answer:

• What four operational questions do only traces answer cleanly?
• Why does a metric that says “checkout is slow” not answer “which call is slow”?
• What is the relationship between the root span’s duration and the sum of its children’s durations?
• Where in the SDK should service.name be set, and why does setting it in the collector fail in a polyglot fleet?
• What is the first attribute to inspect when a known service’s traces do not appear in a TraceQL query?

Quiz