TraceQL Intrinsics

{ trace:duration > 5s } returns the trace IDs of every trace whose wall-clock duration exceeded five seconds. The filter does not depend on any attribute the SDK happens to set; it is a property of the trace itself. This lesson is about those canonical, always-present fields: the intrinsics.

What it is

An intrinsic is a TraceQL field that is part of the span or trace data model rather than part of the attribute map. Intrinsics are written with a scope and a colon: span:status, trace:id, span:duration. The scope tells the engine which level of the trace tree the field belongs to.

The full list of intrinsics supported in TraceQL GA:

  Span-level (apply per span)
  --------------------------
  span:status           enum   ok, error, unset
  span:statusMessage    string  optional text accompanying status
  span:duration         duration end - start of the span
  span:name             string  operation name (e.g. POST /checkout)
  span:kind             enum    server, client, producer, consumer, internal
  span:id               string  span id, hex
  span:parentID         string  parent span id, hex (nil for root)
  span:childCount       integer number of direct children

  Trace-level (apply to the whole trace)
  --------------------------------------
  trace:id              string   trace id, hex
  trace:duration        duration max(end) - min(start) across spans
  trace:rootName        string   name of the root span (if any)
  trace:rootService     string   resource.service.name of the root span

  Event-level (annotations on the span)
  -------------------------------------
  event:name            string   name of the span event
  event:timeSinceStart  duration time of the event relative to span start

  Link-level (causal links to other spans)
  ----------------------------------------
  link:spanID           string   linked span id
  link:traceID          string   linked trace id

  Instrumentation scope (the library that emitted the span)
  ---------------------------------------------------------
  instrumentation:name    string   library name
  instrumentation:version string   library version

Three intrinsics appear without the colon in everyday queries: status, name, duration. They are shorthand for span:status, span:name, span:duration. The colon form is canonical and indexed.

Why a sysadmin cares

Intrinsics are the highest-leverage fields in the language. Five operational reasons they matter:

1. Always present. Every span has a status, a name, a duration, an id, and a parent id — even if the SDK forgot to set a single attribute. A query on an intrinsic returns something; a query on a missing attribute returns nothing.
2. Indexed. Tempo stores intrinsics in dedicated Parquet columns with bloom filters. A filter on span:status = error can skip blocks whose status column contains no error values; a filter on a custom attribute reads every row of that column.
3. Stable across SDK upgrades. A rename of an attribute (http.url to url.full) breaks every selector that uses the old name. span:name and trace:duration are unaffected.
4. Canonical answers to canonical questions. “Is the service returning errors?” is status = error. “How long did the trace take?” is trace:duration. The intrinsics map 1:1 to the operational questions; custom attributes do not.
5. The only way to address the trace structure. Parent, root, childCount, and the trace-level intrinsics (id, duration, rootName, rootService) are the only way to target the shape of the trace tree. A custom attribute cannot tell you whether a span is a leaf.

How it works

Intrinsics are not attributes — they live in dedicated fields of the OTel data model. The Tempo query engine has special handling for them:

  Block in Parquet
       |
       +-- resource columns    (resource.service.name, ...)
       +-- span columns        (span.http.url, span.http.method, ...)
       +-- intrinsic columns   (status, name, duration, kind, ...)
       |
       v
  TraceQL selector
       |
       +-- { span:status = error }        reads the status column (indexed)
       +-- { span.http.status_code = 500 } reads the http.status_code column
       +-- { trace:duration > 5s }        reads the duration column on every span
                                            and computes the trace-level value

A query on an intrinsic reads a column that the engine has indexed specifically for filtering. A query on a custom attribute reads a column the engine has only indexed for storage.

The trace-level intrinsics (trace:id, trace:duration, trace:rootName, trace:rootService) are special: they are identical for every span in the trace. The engine evaluates them once per trace, not once per span. This is the reason a filter on trace:duration > 5s is significantly cheaper than a filter on avg(span:duration) > 5s against the same window.

Under the hood

How to configure it

Intrinsics require no Tempo configuration. They are part of the query language.

The relevant Grafana configuration is the dashboard variable that picks the right scope prefix. A common pattern:

# Grafana dashboard variable
type: query
name: status_intrinsic
datasource:
  type: tempo
  uid: tempo
query:
  qryType: traceqlSearch
  query: '{ span:status != nil }'
  # Optional: restrict the panel to a specific scope prefix.
  options:
    intrinsicScopes: [span, trace]

A panel that uses the variable:

panels:
  - type: traces
    title: 'slow traces'
    datasource:
      type: tempo
      uid: tempo
    targets:
      - query: '{ resource.service.name = "checkout-api" && trace:duration > 2s }'
        queryType: traceql
        limit: 20

How to validate it

Severity: READ-ONLY. Three checks confirm an intrinsic filter is behaving as expected.

1. The intrinsic filter matches what the operator expects:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ trace:duration > 5s }' \
  --data-urlencode 'limit=2' | jq '.traces[].traceID'

"4bf92f3577b34da6a3ce929d0e0e4736"
"7a1f8b9c0d4e3f2a5b6c7d8e9f0a1b2c"

2. The trace actually has the expected duration:

TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | \
  jq '[.resourceSpans[].scopeSpans[].spans[].endTimeUnixNano,
       .resourceSpans[].scopeSpans[].spans[].startTimeUnixNano] | max - min'

The diff is in nanoseconds. Convert to seconds (divide by 1e9) and confirm it is greater than 5.

3. Compare the intrinsic filter against an attribute filter on the same data:

# Intrinsic - fast
curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ trace:duration > 5s }' \
  --data-urlencode 'limit=10' | jq '.traces | length'

# Attribute - slower on the same window
curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ span:duration > 5s } | count() > 0' \
  --data-urlencode 'limit=10' | jq '.traces | length'

The intrinsic returns trace-level matches; the attribute returns spans within traces that contain a single span over 5s. The counts will differ.

How it can fail

Six shapes appear when an intrinsic filter returns the wrong answer:

1. The colon is missing. trace.duration > 5s looks like the intrinsic; the canonical form is trace:duration > 5s (with the colon). Symptom: syntax error in the Tempo UI; the engine does not parse it as an intrinsic.
2. The scope is wrong. span:traceID is not a thing; trace:id is. Symptom: empty result set; the operator assumes the trace has no ID.
3. The status value is wrong. span:status = "error" is a string match; the canonical enum is the unquoted error. Symptom: empty result set; the operator assumes no errors.
4. The duration unit is wrong. trace:duration > 5000 compares 5000 nanoseconds against the trace duration in nanoseconds; the trace is always longer. Symptom: every trace matches; the comparator is a no-op.
5. The structural intrinsic is misunderstood. span:parentID = nil returns root spans (those with no parent). The operator expected it to return “spans whose parent is unset” — which is the same thing, but the intuition is that = nil means “missing”, and root spans have no parent by definition.
6. The intrinsic is filtered on a non-existent scope. event:spanID is not a defined intrinsic. Symptom: syntax error or empty result.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Confirm the syntax. The Tempo UI shows the parsed intrinsic in monospace. If the field is rendered as a plain name rather than a scope-prefixed intrinsic, the colon is missing.
2. Confirm the value type. span:status takes ok, error, unset. span:kind takes server, client, producer, consumer, internal. The Tempo UI lists allowed values; check them.
3. Confirm the duration unit. TraceQL durations are ns, us, ms, s, m, h. A query that uses bare integers compares nanoseconds and matches everything.
4. Confirm the trace is present. Run { } to confirm the time window has any trace; run { trace:rootService = "..." } to confirm the root service is set.
5. Inspect a single trace. Run the trace-by-id endpoint and confirm the field is present.

Security implications

• Trace IDs as identifiers. trace:id returns the trace ID as a string. A query that filters on a known trace ID is effectively a trace-by-id lookup. In multi-tenant mode, the X-Scope-OrgID header still applies; a query that omits it returns unauthorised.
• Status leakage. A query on span:status = error returns spans whose status is error. The status text (span:statusMessage) can contain sensitive error details (a credential in an exception message). Treat the trace backend as sensitive; redact at the SDK.
• No PII on intrinsics. The intrinsics are stable fields of the data model. They do not contain user data, request bodies, or any custom value. PII risk lives on the attribute side, not the intrinsic side.

Performance implications

The cost ranking of intrinsic filters, cheapest to most expensive:

1. Status filter (span:status = error). Indexed bloom filter on a tiny enum column. Sub-second against months of data.
2. Name filter (span:name = "POST /checkout"). Indexed string column with dictionary encoding for repeated names.
3. Kind filter (span:kind = client). Indexed enum column.
4. Duration filter (span:duration > 100ms). Numeric column; no bloom filter; reads every row.
5. Trace-level filter (trace:duration > 5s). Computed once per trace from span start/end; cheaper than per-span duration filters.
6. Structural filter (span:childCount = 0). Reads the span structure; computed per span.

For dashboards that refresh every 30 seconds, prefer trace-level intrinsics and the indexed enum intrinsics. Save per-span duration filters for drill-down queries where the time window is short.

Production guidance

• Always prefer an intrinsic over a custom attribute when the question allows it. The intrinsic is indexed; the attribute is not.
• Use the colon-prefixed form (trace:duration) in production queries. The bare form (duration) parses but is less explicit about the scope.
• Use trace-level intrinsics (trace:duration, trace:rootName, trace:rootService) for cross-service pivots; use span-level intrinsics for span-level pivots.
• Use structural intrinsics (span:parentID, span:childCount) to target the shape of the trace tree — root spans, leaf spans, fan-out detection.

Verification

You should now be able to answer:

• What is the difference between trace:duration and span:duration?
• Why are intrinsics faster than custom-attribute filters?
• Which intrinsic returns only the root span of a trace?
• Why does span:status = "error" (quoted) return nothing while span:status = error (unquoted) works?
• What does span:parentID = nil match?

Quiz