Dependency Attribution

The trace opens. Eighty-one spans laid out across 1.4 seconds. At the top of the trace the root span: POST /checkout, duration 1.4 s. Beneath it, a child span: redis.SET, duration 1.25 s. A handful of microseconds for everything else. The attribution is mechanical: 90% of the request’s latency is the dependency call. The slow downstream is identified. The dependency attribution is the read step — turning a trace timeline into a named cause.

This lesson is the read step. Lessons 03 and 04 are the pivot steps that put the operator in this view; this one is about how to read what is on screen and how to ask Tempo follow-up questions.

What it is

Dependency attribution is the act of naming the dependency span that dominates a service-level latency observation. In a Tempo trace detail panel, attribution is reading the timeline and identifying the longest child span of the root span. In a batch of traces, attribution is a TraceQL query that surfaces the same pattern across many traces.

  Service-level histogram observation
        |
        | pivot from the diamond
        v
  Tempo trace detail panel
        |
        | one pane per service; the root span at the top
        v
  Each child span shows its duration as a horizontal bar
        |
        | the longest bar (after the root) is the dependency attribution
        v
  Read the span attributes: db.system, rpc.service, http.url
  to confirm the dependency name

Two reads:

• Visual. The trace detail panel renders spans as a Gantt-like timeline. The bar widths are proportional to duration. The attribution is the visually dominant child.
• Programmatic. The Tempo API returns the trace as JSON. The attribution is the child span with the largest durationNanos. The dependencies can be confirmed by inspecting the span’s attributes (db.system, rpc.service, http.url, messaging.system).

Why a sysadmin cares

Three situations reduce to dependency attribution:

1. The slow dependency is a single point of failure. The trace timeline shows 1.3 of 1.4 seconds inside rpc.service="payment-svc". The attribution is a single dependency. The fix is upstream.
2. The slow dependency is a chain. The trace timeline shows the root span waiting on cart-svc which waits on db.cart-orders which waits on a Redis BRPOP. The attribution is a chain, not a single dependency. The fix is in the slowest span in the chain — usually the innermost one.
3. The slow dependency varies by trace. No single span dominates every trace; each trace shows a different bottleneck. TraceQL by span attributes returns the distribution and the operator sees the system as a whole rather than chasing the latest individual trace.

The first two are visually obvious on a single trace. The third requires Tempo’s TraceQL — the lesson covers both shapes.

How it works

The trace timeline is a flat list of spans with start and end timestamps. Tempo renders them grouped by service name and ordered within each group by start time. Each span’s durationNanos field is the source of truth for its contribution to the trace’s total duration.

   checkout-svc          |------- POST /checkout (1.4s) --------|
   checkout-svc            |decode| |verify|         |respond|
   payment-svc                             |-- rpc "charge" --| (1.3s)
   payment-svc                              |---| |auth| |---|   |
   db.payments                                          |-- SELECT (1.2s) --|

Three reads from the timeline:

• Root span duration. The total wall-clock duration of the request from the consumer’s perspective.
• Dominant child span. The child span with the longest duration, and the largest fraction of the root.
• Chain attribution. The dominant child may itself have a dominant grandchild. The chain attribution follows the path of longest spans inward until the leaves are reached.

The dominant child is almost always the answer. The chain attribution is the refinement.

How to configure it

Two configurations: the Tempo data source’s derived fields allow the trace panel to link from a span back to other telemetry, and the OTel SDK instrumentation populates the attributes that make spans attributable.

1. Tempo data source — confirm trace-to-logs is configured

The trace detail panel renders the trace; the trace-to-logs link from the span to Loki is the next pivot. Without this, the operator can read the span but cannot jump to the dependency’s own logs.

# Grafana provisioning — Tempo data source (CONFIGURATION)
# Verified on Grafana 11.x.
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      httpMethod: GET
      tracesToLogsV1:
        datasourceUid: loki
        tags: ['job', 'instance', 'service.name']
        mappedTags:
          - key: service.name
            value: service
        spanStartTimeShift: '0s'
        spanEndTimeShift: '0s'
        filterByTraceID: true

This is the inverse of the Internal link from lesson 02. With both directions wired, the operator can move freely between the metric, the trace, and the dependency’s logs.

2. OTel SDK — confirm semantic conventions are honoured

The semantic conventions on the attributes are populated by the client instrumentation libraries. The minimum required for dependency attribution:

# Python — exporting a span with semantic-convention attributes.
# Verified on opentelemetry-instrumentation 0.46b0.
from opentelemetry import trace
from opentelemetry.semconv.trace import SpanAttributes

tracer = trace.get_tracer("checkout")

with tracer.start_as_current_span("charge_payment") as span:
    span.set_attribute(SpanAttributes.RPC_SERVICE, "payment-svc")
    span.set_attribute(SpanAttributes.RPC_SYSTEM, "grpc")
    # The instrumented Postgres client does this automatically;
    # this snippet is the explicit version for a custom dependency.
    span.set_attribute(SpanAttributes.DB_SYSTEM, "postgresql")
    span.set_attribute(SpanAttributes.DB_OPERATION, "SELECT")
    span.set_attribute(SpanAttributes.DB_SQL_TABLE, "cart_orders")
    # perform the call

In most languages the auto-instrumentation library populates these attributes from the client library; manual set_attribute is rarely needed. The configuration question is whether the auto-instrumentation is enabled.

3. Verify the trace has attributes

# READ-ONLY — confirm the trace has the expected attribute set
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '[.traces[0].spans[].attributes[] | select(.key == "rpc.service" or .key == "db.system")]'

Expected: at least one attribute per dependency category. Empty result means the SDK is not populating the attributes and the operator is staring at a timeline with unnamed bars.

How to validate it

Three validation layers.

1. The trace has named dependency spans

# READ-ONLY — confirm at least one span carries rpc.service or db.system
curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '[.traces[0].spans[] | select(.attributes | to_entries | map(.key) | any(. == "rpc.service" or . == "db.system"))] | length'

Expected: at least 1. Zero means the span attributes are absent. The chain attribution cannot be done because there is no way to identify which dependency the slow span represents.

2. Find the dominant span programmatically

# READ-ONLY — find the longest child span for a known root
curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '[.traces[0].spans[] | select(.parentSpanId != null and .parentSpanId != "") | { name, duration: (.durationNanos / 1000000) }] | sort_by(.duration) | reverse | .[0:3]'

Expected: the first entry is the dominant span; its duration in milliseconds is the close to the histogram’s exemplar value (when the bucket is the right one).

3. The trace-to-logs link works

Open the trace in Grafana. Click the dominant span. In the span detail panel, the “Logs for this span” link should resolve and open Loki with the relevant logs filtered by the trace ID. Empty logs page means the trace-to-logs configuration in the Tempo data source is broken or the log records do not carry the trace ID attribute.

How it can fail

Four failure modes, ordered by frequency.

1. Auto-instrumentation is not enabled. The producer emits spans but the spans have no semantic-convention attributes. Symptom: the trace timeline shows the dependency spans; the spans have no rpc.service or db.system labels; the attribution reads as “unknown client” rather than the dependency name.
2. Manual span code wraps a slow operation without setting attributes. A custom dependency call wraps requests.post in tracer.start_as_current_span but does not call set_attribute(SpanAttributes.HTTP_URL, ...). Symptom: the trace timeline shows a long span with a generic name like http_request and no URL.
3. The trace has its dominant child hidden by a misconfigured span kind. A span is marked as span.kind=INTERNAL but the operation crosses service boundaries. Symptom: the trace timeline shows the span at the right time but the auto-grouping doesn’t surface it as a child of the right service.
4. Trace-to-logs is broken. The trace shows the dependency; clicking the dependency span opens an empty Loki panel. Symptom: the trace-to-logs mapping is missing the trace ID attribute or the OTel log SDK pipeline is not attaching the trace ID to log records.

How to troubleshoot it

Steps in order. Each step rules out one of the four failure modes.

1. Check the span attributes via the API. Run the attributes check from validation step 1. If the rpc.service/db.system/http.url set is absent, the auto-instrumentation is not running (failure mode 1) or the manual span wrapping is missing the attribute calls (failure mode 2).
2. Check the span kind via the API. Inspect the dominant span’s kind field. If it is INTERNAL for what should be a CLIENT span, the OTel instrumentation is misnamed.
3. Check the trace-to-logs link. Click the dominant span in Grafana. If the link opens an empty Loki page, inspect the Tempo data source’s tracesToLogsV1 block and the log records’ trace_id attribute. The fix is in the OTel log SDK, not the Tempo data source.
4. Confirm the chain attribution is real. Run the dominant-span query against a known-slow trace. If the dominant span is a chain — its own dominant child is even slower — the attribution is a chain and the deepest slow span is the actual fix location.

Security implications

Dependency attribution reads spans that may carry URL paths, database names, and service identifiers. These are operational metadata, not sensitive data per se, but the aggregation of them is a map of the production architecture. Treat the trace panel as you would a network diagram.

• Spans from internal services are usually fine to expose to all teams; the architecture is not a secret.
• Spans that include URL paths with user IDs (e.g., http.url=/api/users/4242/orders) need redaction. Filter the path through the OTel SDK’s URL sanitiser or remove the sensitive segment at recording time.
• Database names (db.system=postgres, db.namespace=orders_pii) leak naming conventions and sometimes the data subject class. Restrict the trace-readers to teams that need them.

Performance implications

Reading a single trace is cheap. The performance concerns arise when the operator decides to query Tempo for “all traces where payment-svc was slow last hour”:

• { status = error } TraceQL. Bounded by the number of errored spans per service per hour.
• { resource.service.name = "checkout" && duration > 1s }. Returns the trace IDs of all such traces. Tempo executes the query against an iterator. Cost scales with the cardinality of matching spans.
• Trace detail fetch. A single trace’s detail panel cost is bounded by the trace’s size. A 5,000-span trace is a slow render; configure Tempo with max_spans_per_trace and protect the trace pipeline at the SDK level by enforcing a span cap.

Production guidance

• Treat the chain attribution as the primary read. The dominant child of the root span is the starting point. The dominant child of that span is the deepest layer. Three levels is enough; more is custom code that is rarely the bottleneck.
• Capture the OTel auto-instrumentation in CI. A new dependency added to the service without the matching OTel instrumentation produces a trace with an unnamed client span. The CI test should flag new HTTP clients, database drivers, and RPC frameworks without a matching OTel instrumentation.
• Add a TraceQL excerpt to every runbook that names a recurring dependency. The runbook for the payment-svc unavailability should include the TraceQL that surfaces the dependency’s slow traces, not just the trace example.

Verification

You should now be able to answer:

• Which span in a trace timeline is the dependency attribution?
• Which OTel semantic-convention attributes confirm the dependency name?
• How do you follow the chain attribution when the dominant span itself has a slow descendant?
• What is the visible symptom when auto-instrumentation is not running?

Quiz