Trace Exploration Workflows

At 03:14 a latency alert pages the on-call engineer. The dashboard shows POST /checkout p99 at 1.4 s. The engineer follows a six-step flow: pivot to the exemplar trace, read the span timeline, follow the slow leaf to its dependency, open the service map to confirm the dependency is the outlier, drill from the one slow trace to the population of slow traces with a TraceQL filter, and correlate to the matching log line in Loki. Root cause: a connection-pool exhaustion in cart-svc’s Redis client, introduced in the 02:50 deploy. Time-to-root-cause: seven minutes.

This lesson is that flow. It is not a list of features; it is a discipline. The features are the tools; the discipline is the order in which they are pulled.

What it is

A trace exploration workflow is the ordered sequence of queries, pivots, and inspections an operator runs to convert an elevated metric into a verified root cause. The canonical flow has six steps, each one answering the question the previous step left open:

  1. Metric pivot
     Where in the dashboard did the signal come from?
        |
        v
  2. Exemplar click
     Which single trace is the diamond pointing at?
        |
        v
  3. Span timeline
     Which span inside the trace is the slow leaf?
        |
        v
  4. Dependency hop
     Which service owns the slow leaf?
        |
        v
  5. Service map
     Is this one slow request or a pattern across the fleet?
        |
        v
  6. Population filter
     How many other traces share the slow pattern?
        |
        v
  7. Log correlation
     What does the application log say about the request?
        |
        v
  Root cause (verified, with evidence)

Each step has a tool and an output. The output of one step is the input to the next. Skipping a step means carrying the wrong hypothesis into the next one.

Why a sysadmin cares

Three failure shapes appear when the workflow is missing or ad-hoc:

1. The hypothesis with no evidence. An engineer reads the metric, names a probable cause, and starts fixing it. No trace was opened; no log was read. The “fix” deploys at 04:00 and the issue is still there at 04:30. Time-to-resolution doubled because the hypothesis was never tested.
2. The investigation with no end. The engineer opens traces at random, scrolls through them, and never closes the loop. The on-call shift ends; the next engineer picks up the same investigation and starts over. Time-to-resolution is measured in shifts, not minutes.
3. The wall of evidence with no conclusion. Every tool was used; nothing was decided. The incident channel has forty screenshots and one unverified hypothesis. The post-incident review finds that the cause was the second screenshot, not the fortieth.

A documented workflow replaces all three. The output of every step is recorded; the next step is mechanical; the conclusion is forced.

How it works

The six steps in detail:

  Step 1: Metric pivot
  --------------------
  Input:  the alert, the dashboard, the affected metric
  Tool:   Grafana dashboard panel
  Output: the metric and its panel (e.g. histogram p99 of
          POST /checkout)
  Query:  histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{
              route="/checkout"}[5m])) by (le)))

  Step 2: Exemplar click
  ----------------------
  Input:  the histogram panel
  Tool:   Grafana exemplar diamond
  Output: the trace ID of one slow request
  Action: click the diamond; Tempo opens

  Step 3: Span timeline
  ---------------------
  Input:  the trace by ID
  Tool:   Tempo trace view
  Output: the slow leaf span and its parent chain
  Query:  { trace:id = "<id>" }
          -- inspect spans by duration descending

  Step 4: Dependency hop
  ----------------------
  Input:  the slow leaf span
  Tool:   span attributes on the slow leaf
  Output: the downstream service name (resource.service.name
          on the leaf span, or its parent if the leaf is
          the call)
  Query:  { resource.service.name = "<downstream>" }

  Step 5: Service map
  -------------------
  Input:  the entry service and the downstream
  Tool:   Grafana service map view
  Output: confirmation that the downstream is the outlier,
          with a per-edge error rate and latency
  Query:  -- navigate to the service map data source;
          no query required, the map is precomputed

  Step 6: Population filter
  -------------------------
  Input:  the slow trace ID, the slow span name, the slow
          leaf attributes
  Tool:   Tempo TraceQL search
  Output: the count and sample of all traces sharing the
          pattern
  Query:  { name = "<slow span>"
            && resource.service.name = "<downstream>"
            && span:duration > 500ms }
          | count() > 0

  Step 7: Log correlation
  -----------------------
  Input:  the trace ID
  Tool:   Grafana traces-to-logs link
  Output: the application log entries that share the trace ID
  Query:  {service="<downstream>"} |= "<trace_id>"

The flow is iterative: a finding in step 6 may send the engineer back to step 3 with a refined filter. The discipline is to record every step, every query, and every finding.

Under the hood

How to configure it

The workflow requires the same wiring as lesson 05 plus the traces-to-logs link. The minimal Tempo data source config:

# /etc/grafana/provisioning/datasources/tempo.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    access: proxy
    url: http://tempo.internal:3200
    jsonData:
      httpMethod: POST
      # Required for the log-correlation step.
      tracesToLogsV2:
        datasourceUid: loki
        # Loki filter expression template; the trace ID
        # is substituted into the filter at click time.
        filterByTraceID: true
        filterBySpanID: true
        spanStartTimeShift: -1h
        spanEndTimeShift: 1h
      # Required for the service-map step.
      serviceMap:
        datasourceUid: prometheus

Severity: CONFIGURATION. Grafana must be restarted (or the data source reloaded) to apply.

The Prometheus data source needs the exemplar link from lesson 05:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus.internal:9090
    jsonData:
      exemplarTraceIdDestinations:
        - datasourceUid: tempo
          name: traceID
          url: '$${__value.raw}'

How to validate it

Severity: READ-ONLY. Five checks confirm the workflow is wired.

1. The exemplar click opens Tempo. Open a histogram panel with a diamond and click. Confirm Tempo opens to the right trace.

2. The service map renders. Open the “Service Maps” view in Grafana for the entry service. Confirm the downstream edges have latency and error rate annotations.

3. The population filter returns a count. Run the TraceQL query from step 6 against the Tempo API:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ name = "redis-call" && resource.service.name = "cart-svc" && span:duration > 500ms } | count() > 0' \
  --data-urlencode 'limit=5' | jq '.traces | length'

23

Twenty-three traces share the slow pattern. The investigation has a population, not just a sample.

4. The traces-to-logs link returns Loki results. From the trace view in Tempo, click “Logs for this trace”. Confirm the Loki Explore panel opens with the trace ID filter applied.

5. The trace tree is well-formed. Run the trace-by-id endpoint on the slow trace and confirm multiple resource spans (more than one service contributed spans):

TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | \
  jq '[.resourceSpans[].resource.attributes[]
        | select(.key == "service.name") | .value.stringValue]
      | unique'

["api-gateway","checkout-api","cart-svc","payments"]

Four services in the trace tree confirms context propagation is working across the whole path.

How it can fail

Six shapes appear when the workflow stalls:

1. The exemplar is missing. The histogram panel has no diamonds. Cause: Prometheus --enable-feature=exemplar-storage not set; or the SDK does not attach the trace context to observations; or tail sampling dropped the trace.
2. The trace opens but is a single span. Context propagation broke at the first outbound hop. Cause: the SDK is instrumenting the outbound call but not propagating the traceparent header. The fix is in the application, not in Grafana.
3. The slow leaf has no attributes. The span name is generic (http.request) and the attributes are empty. Cause: the SDK is configured with attributes_limit = 0 or a span processor is dropping the map on a size limit.
4. The service map is blank. The metrics-generator is not producing service-graph metrics. Cause: the metrics_generator.processor.service_graphs block is not configured; or the generator is not running; or the metrics it produces are not being scraped.
5. The population filter returns one trace. The filter is too specific. Cause: an attribute that was unique to the sample trace (a request ID, a user ID) leaked into the filter. Loosen the selector by removing the offending attribute.
6. The logs panel is empty. The traces-to-logs link is not configured. Cause: the tracesToLogsV2 block is missing from the Tempo data source, or the Loki data source UID is wrong.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Which step is the operator on? Map the failure to one of the six steps. The fix is local to that step.
2. Is the exemplar present? Hit /api/v1/query_exemplars on the histogram. Empty: the issue is upstream (Prometheus or SDK); the workflow cannot start.
3. Is the trace tree well-formed? Run /api/traces/{id} and count resourceSpans. One service: propagation is broken; the workflow cannot diagnose.
4. Is the slow leaf identifiable? Sort spans by duration. The leaf is at the bottom. A leaf with name = "redis-call" is actionable; a leaf with no name is not.
5. Is the service map rendering? Inspect the traces_service_graph_request_total metric on the metrics-generator. Zero: the generator is not running or is not producing graph metrics.
6. Are the logs correlated? The Loki query {service="<svc>"} |= "<trace_id>" should return at least one line. Empty: the application is not logging the trace ID.

Security implications

• Tenant boundary on the trace view. A multi-tenant Tempo whose X-Scope-OrgID is misconfigured can let an operator in one tenant open traces from another. The trace view must honour the same header as the search endpoint.
• Tenant boundary on the log view. The traces-to-logs link inherits the same boundary. The Loki query that opens from a Tempo click must include the tenant header.
• Sensitive spans. A trace view that includes a span with a credential in the attribute map (an authorisation header, an API key) leaks that credential to anyone who can open the trace. Redact at the SDK.
• Sensitive logs. The log-correlation step inherits the log redaction pipeline. The same fields filtered in the log pipeline must be filtered in the trace attribute map.

Performance implications

• Step 1 (metric). Cheap; the histogram is already aggregated.
• Step 2 (exemplar). Cheap; the exemplar is in Prometheus storage.
• Step 3 (trace by ID). Cheap; one block read.
• Step 4 (dependency hop). Free; the span attributes are already in the trace.
• Step 5 (service map). Cheap; the map is precomputed from the metrics-generator.
• Step 6 (population filter). The expensive step. A TraceQL search against a 30-day window can scan millions of spans. The query-frontend cache helps for repeated dashboard queries but not for one-off investigations. Bound the time window to the smallest range that contains the slow traces.
• Step 7 (log correlation). Cheap; the Loki query is indexed on the trace ID label.

The bottleneck is step 6. The discipline is to design the filter so it can run against the smallest possible window: include the service name, the span name, the duration threshold, and the time range.

Production guidance

• Document the six steps in the team runbook. The order matters; the discipline matters more than the tools.
• Pre-build the service-map view as a dashboard panel that every on-call engineer has bookmarked.
• Configure traces-to-logs and traces-to-metrics links on the Tempo data source. The investigation moves faster when the three signals are joined at the data source level.
• Bound the population filter (step 6) with a limit and a short time window. The slow trace is in the recent past; the population is the same.
• Record every step in the incident channel. The post-incident review needs the evidence trail.

Verification

You should now be able to answer:

• What are the six steps in the canonical trace exploration workflow?
• Which step is the bottleneck for query cost?
• Why must step 6 run before any code change is deployed?
• What is the role of the tracesToLogsV2 block on the Tempo data source?
• What is the most common cause of a blank service map?

Quiz