Trace Exploration

A single trace told you that payment-svc was the slow dependency. The next question is: how many traces tell the same story? Three traces is a coincidence. Thirty traces is a pattern. Three hundred is a regression. Trace exploration is the step that turns anecdote into evidence.

This lesson is about asking Tempo for related traces. The pivot from lesson 02 gave you one. TraceQL gives you many. The discipline is asking narrowly: every query must have a time bound, an attribute filter, and either a duration or a status filter that turns the population into a cohort.

What it is

Trace exploration is the workflow of running TraceQL queries against Tempo to retrieve a set of traces that share an attribute, a duration, or a status. The output is a list of trace IDs and a panel of summary timelines, which the operator can drill into one at a time.

  Single trace the operator is reading
        |
        | identifies a pattern (a slow dependency, an error code, a route)
        v
  Construct a TraceQL query that captures the pattern
        |
        v
  Run the query against Tempo in Explore
        |
        v
  Result is a list of matching traces
        |
        v
  Triage: pick a few, read the detail panels, confirm the pattern

TraceQL is a query language, not a script. Each query is a single expression with predicates joined by && and ||. The predicates address either the trace as a whole (intrinsic fields like status and duration) or the spans inside the trace (intrinsic fields plus the structured attribute set).

Why a sysadmin cares

Three situations reduce to trace exploration:

1. Verifying a fix. A slow dependency has been changed and the team wants to know whether the same TraceQL that surfaced the issue ten minutes ago now returns zero traces. The answer is “fix is live” or “fix did not take”.
2. Triaging a regression. A deployment at 14:20 may have introduced a regression. TraceQL by service.version returns the traces whose version is the new release; the operator compares the latency distribution against the previous version.
3. Capacity planning. TraceQL by customer.id returns the traces by customer; the operator identifies the top-N customers by span count or duration and knows where capacity is going.

In all three cases, the operator opens the trace exploration panel, runs a TraceQL, and reads the result. The discipline is keeping the queries bounded — the next section covers that.

How it works

TraceQL has two layers:

  Trace-level filter  -> selects traces, not spans
                          e.g., { duration > 1s }
                          e.g., { status = error }
                          e.g., { rootName = "POST /checkout" }

  Span-level filter   -> selects traces by what their
                          spans contain
                          e.g., { span.db.system = "postgresql" }
                          e.g., { span.http.status_code = 500 }
                          e.g., { span.rpc.service = "payment-svc" && span.duration > 1s }

A query with only a trace-level filter is the strict pattern: every selected trace has the property everywhere on the root span. A query with a span. filter is the looser pattern: the trace is selected when any span in the trace matches. Both are useful; the strict filter is cheaper for Tempo to evaluate.

Tempo evaluates TraceQL by streaming through the trace-by-trace index of recently ingested traces. The result set is a list of trace IDs that match the predicates. The operator sees those trace IDs as a list in Grafana and can open any one for the full detail.

How to configure it

Two configuration steps.

1. Confirm the Tempo data source is provisioned for TraceQL

TraceQL is the default query language for Tempo in Grafana 11.x. No specific configuration is required to enable it; the data source must be reachable.

# Grafana provisioning — Tempo data source (CONFIGURATION)
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    access: proxy
    jsonData:
      httpMethod: GET
      tracesToLogsV1:
        datasourceUid: loki
        tags: ['job', 'instance', 'service.name']

The relevant panel type in Grafana 11.x is “TraceQL query” under the Tempo data source. The query editor accepts the expression and the time range directly.

2. Build the TraceQL query panel

The canonical TraceQL exploration panel is a Grafana Explore view with a query editor that supports the <datasource>.queryType of traceql. The minimal configuration:

# Panel options (CONFIGURATION)
panel:
  type: traces
  title: 'slow payment-svc spans — last 10 minutes'
  datasource: tempo
  targets:
    - refId: A
      queryType: traceql
      query: '{ resource.service.name = "checkout" && span.rpc.service = "payment-svc" && duration > 1s }'
      limit: 20

The limit field caps the result set. Setting it to 20 means the panel shows up to 20 trace IDs at once; the operator can scroll further into the result set with the pagination control.

A more refined version narrows by time and version:

{ resource.service.name = "checkout" && span.rpc.service = "payment-svc" && duration > 1s && resource.service.version = "2026.08.13-abc123" }

The query returns traces whose root service is checkout, that have a payment-svc span, that take over a second, that match the version released today. This is the canonical regression investigation pattern.

3. Verify the panel query is bounded

# READ-ONLY — exercise the TraceQL via the Tempo API directly
# This is what the Grafana panel does under the hood.
curl -sfG http://tempo:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" && span.rpc.service = "payment-svc" && duration > 1s }' \
  --data-urlencode 'limit=20' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.traces | length'

Expected: a small positive integer. Larger values (hundreds) mean the query is too broad — narrow with another attribute or shorten the time range.

How to validate it

Three validation layers.

1. The TraceQL returns a non-empty result for the cohort

# READ-ONLY — check at least one match in the time window
curl -sfG http://tempo:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" && span.rpc.service = "payment-svc" }' \
  --data-urlencode 'limit=10' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.traces | length'

Expected: a positive integer when the dependency call happens during the window. Zero means either the window is wrong or the service has not emitted traces for the cohort.

2. The duration filter applies

# READ-ONLY — confirm the duration filter narrows the result
LARGE=$(curl -sfG http://tempo:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" && span.rpc.service = "payment-svc" }' \
  --data-urlencode 'limit=100' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.traces | length')

LARGE_AND_SLOW=$(curl -sfG http://tempo:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout" && span.rpc.service = "payment-svc" && duration > 1s }' \
  --data-urlencode 'limit=100' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.traces | length')

[ "$LARGE" -ge "$LARGE_AND_SLOW" ] && echo "filter applies"

Expected: the count with the duration filter is less than or equal to the count without it. If LARGE_AND_SLOW is greater than LARGE, the indexes are stale or the predicate is being interpreted as inclusive of unrelated traces.

3. The Grafana panel renders the result

Open Explore in Grafana, switch to the Tempo data source, enter the TraceQL, and confirm the panel shows trace IDs. A blank result is the same as the API returning zero. The panel’s limit should match the API’s limit for consistency.

How it can fail

Four failure modes, ordered by frequency.

1. Query is unbounded in time. The query spans a long window. Tempo returns a large result set and the panel times out. Symptom: the Tempo UI spins, then returns a 504 Gateway Timeout or a “rate limited” error.
2. Query has no indexed predicate. A query like { span.db.statement = "SELECT ..." } is full-trace scan with no way to cut the result down. Symptom: Tempo consumes CPU but the result never returns within the timeout.
3. Span attribute is missing from the trace. The producer does not populate the OTel semantic-convention attribute the query filters on. Symptom: the query returns zero results even though traces exist that visibly match the pattern.
4. TraceQL function not supported by the Tempo version. Some advanced functions (count_over_time, compare) require specific Tempo builds. Symptom: the panel returns a “function not available” or 400 error.

How to troubleshoot it

Steps in order. Each step rules out one of the four failure modes.

1. Check the time window. Confirm the panel’s time range is bounded. A common bug is a panel left at “Last 30 days” when the question is about a deployment. Tighten the range; re-run.
2. Check the indexed-predicate position. Add an indexed predicate (status = error, duration > 1s, rootName = "POST /checkout") at the start of the query. Tempo’s TraceQL engine evaluates indexed predicates first; loose predicates later.
3. Inspect the attribute on a known trace. Pull one trace via the pivot from lesson 02 and verify the attribute is populated on the spans the query is meant to filter on. Empty attribute means failure mode 3 — fix the producer.
4. Check the Tempo version and feature flags. Some advanced TraceQL functions require Tempo 2.4 or later. Validate against the local Tempo binary’s /api/status/version.

Security implications

TraceQL can return spans that carry URL paths, user IDs, and request bodies. The pivot from lesson 02 carries the same risks; the exploration step carries them at a larger scale.

• Cross-tenant queries. Tempo enforces per-tenant isolation in multi-tenant deployments. Verify the operator’s Grafana role maps to the right tenant.
• Service-version leakage. A query by resource.service.version exposes release tagging. Treat it as low-grade metadata.
• PII redaction. Spans with PII in attributes (e.g., enduser.id, http.url with embedded user IDs) need sanitisation at the producer. TraceQL does not redact.

Performance implications

• Per-query cost is O(N traces in window) for indexed predicates. The cost grows linearly with the trace count in the chosen window; the panel limit does not bound this cost, only the rendered result size.
• Per-span cost is O(spans-per-trace) for non-indexed predicates. Avoid predicates that scan every span. Prefer predicates on intrinsic fields plus a small set of resource attributes.
• Panel polling. A Grafana Explore panel that polls Tempo every 5 seconds runs the query every 5 seconds. A six-hour window with a 5-second poll multiplies Tempo’s load by 4,320. Bound the time range.

Production guidance

The four discipline rules that keep TraceQL bounded:

1. Always include a time range. The panel’s time range must be explicit. “Last 30 days” is rarely the right answer.
2. Always include at least one indexed predicate. status, duration, or rootName should be the first clause.
3. Bound limit to a small number. 20 traces is enough for triage. The pagination control can pull more if needed.
4. Bind by resource attribute when possible. resource.service.name is a more selective predicate than any span-level attribute.

A panel that follows all four is safe to embed in a dashboard. A panel that drops any one of them should not be embedded; it is an ad-hoc Explore view only.

Verification

You should now be able to answer:

• Which three predicates are intrinsically indexed in Tempo’s TraceQL engine?
• How does a span-level filter (span.) differ from a trace-level filter in terms of cost?
• What are the four discipline rules that keep TraceQL bounded?

Quiz