TraceQL Selectors

{ span.http.status_code = 500 } selects every span whose HTTP status is 500. Wrapped in a TraceQL pipeline, it returns the trace IDs whose checkout path produced a 5xx response. That single selector is the difference between “the wall of every trace” and “the twelve traces that matter”.

This lesson is about selectors: the condition inside the curly braces, the comparison operators that connect them, and the scopes that decide which attribute the engine reads.

What it is

A TraceQL selector is a filter expression over one span at a time. It is the contents of a pair of curly braces, optionally combined with other selectors using the logical operators && (and) and || (or).

The grammar of a selector has three parts:

selector = scope "." name comparator value
         | intrinsic comparator value

scope      = "span" | "resource" | "event" | "link" | "instrumentation"
intrinsic  = "span:status" | "span:duration" | "span:name" | ...
comparator = "=" | "!=" | ">" | ">=" | "<" | "<=" | "=~" | "!~"
value      = integer | duration | float | string | nil

A selector can be combined with another selector using a logical operator. The whole expression is evaluated per span; if true, the span survives the selector; if false, the span is discarded.

{ resource.service.name = "checkout-api" && status = error }
{ span.http.status_code >= 500 || span.http.status_code = 429 }
{ name = "GET /api/products" && span.http.method = "GET" }
{ span.http.url =~ ".*/v2/orders.*" }

The first example selects every error span from checkout-api. The second selects spans that returned either 5xx or 429. The third combines a span name with a method. The fourth uses a regex on the URL.

Why a sysadmin cares

A selector is the lever the operator pulls to reduce a trace population to the population that matters. Three operational realities make selectors indispensable:

1. The cardinality of “every trace” is too high. A service producing 100 spans per request and 1,000 requests per second produces 100,000 spans per second. Without a selector, the Tempo UI returns a few thousand traces at random, sorted by recency, none of which are the ones the operator needs.
2. The trace is rarely the answer. The answer is the span inside the trace — the slow leaf, the failing database call, the timed-out HTTP client. Selectors locate spans inside traces; from there the engineer drills into the full tree.
3. Selectors survive across schemas. A filter on status = error keeps working when the SDK renames every HTTP attribute from http.url to url.full. A filter on the renamed attribute breaks. Choosing intrinsics when they are available insulates dashboards from SDK churn.

How it works

The engine evaluates a selector against each span in each block that intersects the time window. Three steps, per block:

  Block of spans in time window
            |
            v
  For each span in the block:
            |
            +-- Read the columns the selector touches
            |     (only the span.http.status_code column for the
            |      selector { span.http.status_code = 500 })
            |
            +-- Apply the comparator
            |     (does the read value equal the literal 500?)
            |
            +-- Combine with && and || across the selector
                  (true if every condition matches)
            |
            v
  Span survives if the selector is true; discarded otherwise

The result is a list of trace IDs whose tree contained at least one surviving span.

Scopes

Every attribute lives under a scope. The scope is part of the attribute name and is mandatory for non-intrinsic fields:

Intrinsic fields (no scope) are canonical names for properties the OTel data model defines. They have a scope-prefixed form too: status is shorthand for span:status; name is shorthand for span:name; duration is shorthand for span:duration. The scope-prefixed form is the indexed form and is what the engine reads first.

Comparison operators

Eight operators cover the cases a production selector needs:

Regexes in TraceQL use Go regex syntax and are fully anchored at both ends. The selector span.http.url =~ "/v2/orders" matches exactly that string. To match a substring, anchor with .*:

# Wrong - anchored, only matches the exact string "/v2/orders"
{ span.http.url =~ "/v2/orders" }

# Right - matches any URL containing /v2/orders
{ span.http.url =~ ".*/v2/orders.*" }

Value types

TraceQL distinguishes four value types. The right type matters; the wrong type returns zero results without warning.

A common production mistake is comparing an integer attribute to a quoted string:

# Wrong - 500 is the string "500", not the integer 500
{ span.http.status_code = "500" }

# Right
{ span.http.status_code = 500 }

The first matches nothing if the attribute is an integer; the second matches the intended spans.

Combining selectors

Two selectors in the same braces use &&. Each && requires the span to satisfy both conditions:

{ resource.service.name = "checkout-api" && status = error }

Two braces separated by && is a spanset operator and means something different: “the trace contains spans satisfying the left selector AND spans satisfying the right selector” (they can be different spans):

# Two-condition selector: same span must match both
{ span.http.method = "DELETE" && status != ok }

# Spanset operator: trace has a DELETE method somewhere AND a non-ok status somewhere
{ span.http.method = "DELETE" } && { status != ok }

The single-brace form is stricter. The double-brace form lets the engine find the slow dependency in a trace where the root span is 200 OK but a child span is 500.

Under the hood

How to configure it

Selectors are query-side, not configuration-side. The Tempo configuration that affects selector performance is the same as in lesson 01: querier.max_query_length caps the window, query_frontend.split_queries_by_interval parallelises the scan, and the storage backend is the source of the columns.

The Grafana side is a Tempo data source with a query editor that parses TraceQL and shows syntax errors inline. The editor accepts the full grammar; a typical dashboard variable that feeds a panel looks like:

# Grafana dashboard variable
type: query
name: service
datasource:
  type: tempo
  uid: tempo
query:
  qryType: traceqlSearch
  query: '{ resource.service.name != nil }'

A panel that uses the variable:

panels:
  - type: traces
    title: 'checkout-api errors'
    datasource:
      type: tempo
      uid: tempo
    targets:
      - query: '{ resource.service.name = "$service" && status = error }'
        queryType: traceql
        limit: 20

How to validate it

Severity: READ-ONLY. Three checks confirm a selector is behaving as expected.

1. The selector matches what the operator expects. Run it with a small limit and inspect one trace:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ resource.service.name = "checkout-api" && status = error }' \
  --data-urlencode 'limit=2' | jq '.traces[].traceID'

# "4bf92f3577b34da6a3ce929d0e0e4736"
# "7a1f8b9c0d4e3f2a5b6c7d8e9f0a1b2c"

2. The trace the engine returned actually contains a span with status = error:

TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736
curl -s "http://tempo.internal:3200/api/traces/${TRACE_ID}" | \
  jq '.resourceSpans[].scopeSpans[].spans[]
        | select(.status.code == "STATUS_CODE_ERROR")
        | .name'

# "payment-charge"

A non-empty list confirms the selector and the engine agree.

3. The selector returns a different result set than the empty selector ({ }). This proves the selector is doing work:

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ }' \
  --data-urlencode 'limit=5' | jq '.traces | length'
# 5

curl -sG http://tempo.internal:3200/api/search \
  --data-urlencode 'query={ status = error }' \
  --data-urlencode 'limit=5' | jq '.traces | length'
# 2

Two traces (errors) versus five (everything) shows the selector is filtering.

How it can fail

Six shapes appear when a selector returns the wrong answer:

1. The scope is wrong. The operator writes span.service.name = "checkout-api" instead of resource.service.name = "checkout-api". service.name is a resource attribute, not a span attribute. Symptom: empty result set; the engineer concludes the service has no traces.
2. The attribute name is misspelled. http.status.code is not a thing; http.response.status_code or http.status_code is (depending on SDK version). Symptom: empty result set; the operator assumes tracing is broken.
3. The integer is quoted. { span.http.status_code = "500" } compares the integer 500 to the string “500”. Symptom: empty result set; the operator assumes no 5xx errors are happening.
4. The regex is unanchored. { span.http.url =~ "v2" } is anchored and matches only the literal string “v2”. Symptom: the operator expects substring matching, gets nothing, and assumes the URL pattern does not exist.
5. The value type is wrong. { span.http.status_code > 5 } compares as an integer; { span.http.status_code > "5" } compares as a string (lexicographic). Symptom: a selector that “works” returns a different set than expected.
6. The selector is right but the time window is wrong. The traces exist but in a different window. Symptom: empty result set despite healthy traces. The fix is on the Grafana time picker, not the selector.

How to troubleshoot it

Ordered diagnostics, cheapest first:

1. Confirm the time window has traces. Run { } against the same window; if it returns zero, the issue is retention or clock skew.
2. Confirm the scope and name. Run the selector against a single known trace ID and inspect the span resource / span attribute map. The exact key in the JSON must match the scope and name in the selector.
3. Loosen the selector one condition at a time. Start with { }, add resource.service.name = "...", add status = error. The first version that returns the wrong set pinpoints the broken condition.
4. Quote or unquote the value. An integer that is quoted returns nothing; a string that is not quoted returns nothing. The Tempo UI shows the parsed type next to the value.
5. Test the regex on regex101.com. TraceQL uses Go regex syntax. A regex that “looks right” can fail to compile or match nothing because of anchoring.

Security implications

TraceQL selectors are structured; an attacker cannot break out of the syntax through attribute values. The relevant surfaces:

• Tenant boundary. A Grafana data source with the wrong X-Scope-OrgID sends the selector against the wrong tenant’s data. Symptom: a viewer-role user sees traces from another team.
• Sensitive attributes. A selector on event.exception.message can return spans whose exception message contains a credential. Treat the trace backend as sensitive; redact at the SDK before export.
• Result disclosure. A selector that returns a large set is not a denial of service, but it can be a slow query that saturates the querier. Bound every panel with a limit.

Performance implications

The cost of a selector is dominated by:

• Cardinality of the attribute. A low-cardinality attribute (status, http.method, db.system) is cheap because the column is dictionary-encoded. A high-cardinality attribute (span.http.url, user_id) is expensive because every row must be compared.
• Type of the comparator. Equality (=) is the cheapest. Inequality (!=) scans every row that does not match. Regex (=~) compiles the pattern and scans every row.
• Number of conditions. Each && adds a column read. A selector with three conditions reads three columns; one with five reads five.
• Time window. A 30-day window is 60x more expensive than a 12-hour window for the same selector.

The rule of thumb: a single equality on a low-cardinality intrinsic is sub-second against months of data; a regex on a high-cardinality span attribute is seconds against minutes of data.

Production guidance

• Prefer intrinsics (status, name, duration) over custom-attribute selectors when the question allows it. Intrinsics are indexed and faster.
• Use resource. scopes for service-level pivots; span. scopes for span-level pivots; reserve event. and link. for fine-grained questions.
• Quote regex patterns with .* on both ends. Anchoring is the default and surprises operators who expect substring matching.
• Bound every selector with a limit in Grafana and a max_query_length on the querier.

Verification

You should now be able to answer:

• What three scopes does a TraceQL selector read attributes from, and which one carries service.name?
• What is the difference between = and =~ in a TraceQL selector?
• Why does a regex match nothing when written without .*?
• What is the difference between { A && B } and { A } && { B }?
• What is the first thing to check when a new selector returns zero results?

Quiz