Lab: Query Traces with TraceQL

Objective

Every TraceQL tutorial has the same flaw: it runs queries against data nobody counted, so a result of “seven traces” is unfalsifiable. If the query is wrong you get seven of the wrong traces and no way to tell.

This lab removes that. You will seed Tempo with exactly twenty-four traces in four shapes you defined, so every query has a right answer you worked out in advance. Then you will write the queries that answer real on-call questions, and reproduce — deliberately — the four ways a TraceQL selector returns nothing while looking perfectly reasonable.

Architecture

One Tempo, one seed script, one shell.

   seed-traces.sh                 query helpers (tq / n)
     |  24 POSTs of OTLP/JSON       |  GET /api/search?q=...
     v                              v
  +--------------------------------------------------+
  |  tempo (monolithic)  127.0.0.1:3200  127.0.0.1:4318 |
  |    blocks + WAL on the tempo-data volume            |
  +--------------------------------------------------+

  The corpus, by shape:

    12 x checkout, fast     root 80 ms   payment 45 ms   all OK
     4 x checkout, slow     root 1500 ms payment 1400 ms all OK
     3 x checkout, failed   root 210 ms  payment 150 ms  payment span = error
     5 x inventory          root 60 ms   two spans, no payment service

    Every checkout trace has 4 spans across 2 services.
    Every inventory trace has 2 spans across 1 service.

The three checkout shapes are chosen to defeat lazy queries. The failed traces have a healthy root span and an error on a child two levels down, which is the single most common real trace shape and the one that a root-span-only filter misses entirely. The slow traces are slow in the child, not in the root’s own work. And the inventory traces exist so that trace:rootService has more than one value and a query that forgets to scope by service is visibly wrong.

Requirements

A disposable Linux host with Docker Engine and the Compose v2 plugin, or a Tempo you already control. If you still have the stack from the Deploy Tempo lab, skip Task 1 and point TEMPO at it.
Ports 3200 and 4318 free on loopback, if Task 1 is starting Tempo.
curl, jq, od and GNU date. The seed script needs nanosecond timestamps, which is date +%s%N and therefore GNU coreutils.
About 2 GB of free disk for the image and the volume.
No out-of-band access requirement. Nothing here touches the host network configuration; every published port is bound to 127.0.0.1.

Scenario

An alert fired at 02:40: checkout p99 latency crossed two seconds. The metrics dashboard shows the spike and nothing else — the rate is normal, the error ratio is normal, and the four downstream services all look healthy on their own panels. The trace store has the answer, and the only thing standing between the on-call engineer and it is the ability to ask a specific question.

The questions they actually need answered, in order:

Are there any traces at all in this window?
Which traces took longer than a second?
Which traces contain a failure anywhere, not just at the entry point?
Is the slow part the entry service’s own work, or something it called?
How many spans does a normal trace have, and do the slow ones have more?
Which of these queries can I afford to put on a dashboard that refreshes every thirty seconds?

This lab answers all six against a corpus you can verify by hand.

Tasks

Task 1: Stand up a Tempo, or point at one you have

Skip to Task 2 if you already have one.

LAB="$HOME/traceql-lab"
mkdir -p "$LAB"
cd "$LAB"

ss -ltn 2>/dev/null | grep -E ':(3200|4318)' || echo "both lab ports are free"

curl -s 'https://hub.docker.com/v2/repositories/grafana/tempo/tags?page_size=25' \
  | jq -r '.results[].name' | grep -E '^[0-9]+\.[0-9]+' | head

# Substitute a tag the command above printed:
TEMPO_TAG=2.6.1
echo "TEMPO_TAG=$TEMPO_TAG" > .env

cd "$HOME/traceql-lab"

cat > docker-compose.yaml <<'YAML'
services:
  tempo-init:
    image: grafana/tempo:${TEMPO_TAG}
    user: root
    entrypoint: ["chown", "-R", "10001:10001", "/var/tempo"]
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:${TEMPO_TAG}
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "127.0.0.1:3200:3200"
      - "127.0.0.1:4318:4318"
    restart: unless-stopped

volumes:
  tempo-data:
YAML

cat > tempo.yaml <<'YAML'
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

ingester:
  # Short so the corpus becomes searchable inside the lab session. The
  # production default is far larger, and the wait before a search returns
  # anything is derived from this number rather than guessed.
  max_block_duration: 1m

compactor:
  compaction:
    block_retention: 24h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal
YAML

docker compose up -d
for i in $(seq 1 30); do curl -fsS http://127.0.0.1:3200/ready && break; sleep 2; done

export TEMPO="http://127.0.0.1:3200"
export OTLP_HTTP="http://127.0.0.1:4318"

Task 2: Seed a corpus you counted

The generator below emits OTLP over HTTP in its JSON encoding, so there is no SDK or agent between you and the data. Everything about each trace — the span names, the parent links, the durations, the status codes — is chosen here, which is what makes the rest of the lab checkable.

cd "$HOME/traceql-lab"

cat > seed-traces.sh <<'SH'
#!/usr/bin/env bash
# Emit a trace corpus with known contents.
#   12 fast checkout, 4 slow checkout, 3 failed checkout, 5 inventory.
set -euo pipefail

OTLP="${OTLP_HTTP:-http://127.0.0.1:4318}/v1/traces"

hex() { od -An -N"$1" -tx1 /dev/urandom | tr -d ' \n'; }
astr() { printf '{"key":"%s","value":{"stringValue":"%s"}}' "$1" "$2"; }
aint() { printf '{"key":"%s","value":{"intValue":"%s"}}' "$1" "$2"; }

# name kind traceId spanId parentId startNs durationMs statusCode attrsJson
span() {
  printf '{"traceId":"%s","spanId":"%s","parentSpanId":"%s","name":"%s",' \
    "$3" "$4" "$5" "$1"
  printf '"kind":%s,"startTimeUnixNano":"%s","endTimeUnixNano":"%s",' \
    "$2" "$6" "$(( $6 + $7 * 1000000 ))"
  printf '"status":{"code":%s},"attributes":[%s]}' "$8" "$9"
}

# serviceName spansJson
res() {
  printf '{"resource":{"attributes":[%s,%s]},' \
    "$(astr service.name "$1")" "$(astr deployment.environment lab)"
  printf '"scopeSpans":[{"scope":{"name":"runbook-academy.traceql-lab"},'
  printf '"spans":[%s]}]}' "$2"
}

post() {
  curl -s -o /dev/null -w '%{http_code} ' -X POST "$OTLP" \
    -H 'Content-Type: application/json' --data-binary "{\"resourceSpans\":[$1]}"
}

# rootMs payMs dbMs httpStatus paySpanStatus ageSeconds
checkout_trace() {
  local root_ms=$1 pay_ms=$2 db_ms=$3 http=$4 payst=$5 age=$6
  local tid sa sb sc sd t0 sess
  tid=$(hex 16); sa=$(hex 8); sb=$(hex 8); sc=$(hex 8); sd=$(hex 8)
  sess=$(hex 8)
  t0=$(( $(date +%s%N) - age * 1000000000 ))

  local front back
  front="$(span 'POST /checkout' 2 "$tid" "$sa" '' "$t0" "$root_ms" 1 \
      "$(astr http.request.method POST),$(astr http.route /checkout),\
$(aint http.response.status_code 200),$(astr session.id "$sess")"),"
  front+="$(span 'GET /payments/authorize' 3 "$tid" "$sb" "$sa" \
      "$(( t0 + 10000000 ))" "$(( root_ms - 20 ))" 1 \
      "$(astr http.request.method GET),$(astr server.address payment-svc)")"

  back="$(span 'POST /authorize' 2 "$tid" "$sc" "$sb" \
      "$(( t0 + 20000000 ))" "$pay_ms" "$payst" \
      "$(astr http.request.method POST),$(astr http.route /authorize),\
$(aint http.response.status_code "$http")"),"
  back+="$(span 'SELECT payments' 3 "$tid" "$sd" "$sc" \
      "$(( t0 + 30000000 ))" "$db_ms" 1 \
      "$(astr db.system postgresql),$(astr db.namespace payments)")"

  post "$(res checkout-api "$front"),$(res payment-svc "$back")"
}

# ageSeconds
inventory_trace() {
  local age=$1 tid sa sb t0
  tid=$(hex 16); sa=$(hex 8); sb=$(hex 8)
  t0=$(( $(date +%s%N) - age * 1000000000 ))
  local spans
  spans="$(span 'GET /stock' 2 "$tid" "$sa" '' "$t0" 60 1 \
      "$(astr http.request.method GET),$(astr http.route /stock),\
$(aint http.response.status_code 200)"),"
  spans+="$(span 'SELECT stock' 3 "$tid" "$sb" "$sa" \
      "$(( t0 + 5000000 ))" 15 1 \
      "$(astr db.system postgresql),$(astr db.namespace inventory)")"
  post "$(res inventory-api "$spans")"
}

echo "12 fast checkout traces"
for i in $(seq 1 12); do checkout_trace 80 45 12 200 1 $(( i * 5 )); done; echo

echo "4 slow checkout traces"
for i in $(seq 1 4); do checkout_trace 1500 1400 1300 200 1 $(( i * 7 )); done; echo

echo "3 failed checkout traces"
for i in $(seq 1 3); do checkout_trace 210 150 20 500 2 $(( i * 11 )); done; echo

echo "5 inventory traces"
for i in $(seq 1 5); do inventory_trace $(( i * 9 )); done; echo
SH

chmod +x seed-traces.sh

Configuration changelab host— Writes 24 traces into Tempo. Prints one HTTP status per POST.

$ ./seed-traces.sh

Every response should be 200. A 400 means the payload was rejected; the first thing to check is that date +%s%N prints digits rather than a trailing letter N, because a non-numeric timestamp fails validation for every span in the batch.

Wait for a block to be cut before searching. By-ID lookups work immediately; search does not, because search runs against blocks:

sleep 90

Why the corpus is emitted as two resources per trace

A trace in OTLP is not a single object. It is a set of spans that share a trace ID, and the wire format groups them by the resource that produced them. The resourceSpans array in each POST here has two entries — one for checkout-api and one for payment-svc — carrying two spans each.

That is why resource.service.name is a resource scope in TraceQL and not a span attribute: it is attached one level up, to the emitting process, not to each individual span. A selector written as span.service.name reads a span attribute that does not exist and matches nothing, which is Task 5’s first trap and the single most common TraceQL mistake in production.

It is also why the parent link crosses the resource boundary cleanly: span sc on payment-svc names span sb on checkout-api as its parent. Nothing about the resource grouping affects the tree; the tree lives entirely in the span and parent IDs.

Task 3: Set the window, and build two helpers

Nearly every “TraceQL returns nothing” report is a window problem. Fix the window once, explicitly, and take it out of the variables:

export TEMPO="${TEMPO:-http://127.0.0.1:3200}"
export START="$(date -d '-30 min' +%s)"
export END="$(date +%s)"

# Run a TraceQL search. $1 is the query, $2 an optional limit.
tq() {
  curl -fsSG "$TEMPO/api/search" \
    --data-urlencode "q=$1" \
    --data-urlencode "start=$START" \
    --data-urlencode "end=$END" \
    --data-urlencode "limit=${2:-100}"
}

# Count the traces a query returns.
n() { tq "$1" "${2:-100}" | jq '.traces | length'; }

start and end are unix seconds. They are not optional in practice: leave them out and you get whatever default window the build uses, which is the difference between an empty result and a correct one for reasons that have nothing to do with your query.

Confirm the corpus landed, and learn what the store thinks it has:

n '{}'                                    # expect 24

curl -fsS "$TEMPO/api/search/tags" | jq -r '.tagNames[]' | sort | head -20
curl -fsSG "$TEMPO/api/search/tag/service.name/values" | jq -r '.tagValues[]'

Those are the v1 tag endpoints, which report attribute names without their scope. Newer builds also serve /api/v2/search/tags, which groups the same names under resource, span and the intrinsics — worth preferring when it is available, because the scope is exactly the thing Trap 1 below gets wrong.

The tag-discovery endpoints are the answer to “what can I even filter on here”. Use them before writing a selector against a service you did not instrument yourself — guessing an attribute name and getting zero results is indistinguishable from the service having no traces, and that ambiguity has cost more incident minutes than any other TraceQL behaviour.

Task 4: Answer the six questions

# 1. Anything at all in the window?
n '{}'                                                   # expect 24

# 2. Which traces took longer than a second, end to end?
n '{ trace:duration > 1s }'                              # expect 4

# 3. Which traces contain a failure anywhere in the tree?
n '{ span:status = error }'                              # expect 3

# 4. Which of those failures are invisible from the entry point?
n '{ resource.service.name = "checkout-api" && span:status = error }'
# expect 0 - the checkout spans are all OK; the error is downstream

n '{ resource.service.name = "payment-svc" && span:status = error }'
# expect 3

# 5. How many spans does a normal trace have?
n '{} | count() > 3'                                     # expect 19

# 6. Which traces entered through which service?
n '{ trace:rootService = "checkout-api" }'               # expect 19
n '{ trace:rootService = "inventory-api" }'              # expect 5

Question 4 is the one worth pausing on. The three failed traces return a healthy 200 at the entry point and carry the error two levels down, which is exactly what a retry, a fallback, or a fire-and-forget call produces in production. A dashboard panel filtered on the entry service’s own status is green throughout the incident. This is not a contrived case — it is the default shape of a failure in any system with a resilience layer.

Task 5: Reproduce the four silent zeros

A TraceQL selector that is wrong does not error. It returns an empty list, which looks exactly like “there is no such data”. Produce all four, with the corpus as the control:

# Trap 1 - wrong scope. service.name lives on the resource, not the span.
n '{ span.service.name = "payment-svc" }'                # 0  (wrong)
n '{ resource.service.name = "payment-svc" }'            # 19 (right)

# Trap 2 - quoted integer. The attribute is an int; "500" is a string.
n '{ span.http.response.status_code = "500" }'           # 0  (wrong)
n '{ span.http.response.status_code = 500 }'             # 3  (right)

# Trap 3 - unanchored regex. TraceQL anchors at both ends.
n '{ resource.service.name =~ "payment" }'               # 0  (wrong)
n '{ resource.service.name =~ ".*payment.*" }'           # 19 (right)

# Trap 4 - the window. The query is fine; the window is not.
OLD_START=$START
START="$(date -d '-30 days' +%s)"; END="$(date -d '-29 days' +%s)"
n '{}'                                                   # 0  (wrong window)
START=$OLD_START; END="$(date +%s)"
n '{}'                                                   # 24 (right)

Four different bugs, one indistinguishable symptom. That is why the troubleshooting order in the selectors lesson starts with the bare {} matcher: it separates “this window has no data” from “my selector is wrong” in a single query, and it is the only step that can be done before you know anything about the schema.

Task 6: Ask about the shape of the tree

Two selectors in one pair of braces describe one span. Two pairs of braces describe one trace. The difference is not stylistic:

# One span must be both from payment-svc and an error.
n '{ resource.service.name = "payment-svc" && span:status = error }'   # 3

# The trace must contain a checkout span somewhere AND an error somewhere.
# They can be, and here are, different spans.
n '{ resource.service.name = "checkout-api" } && { span:status = error }'   # 3

# The same question, with the single-brace form: no span is both.
n '{ resource.service.name = "checkout-api" && span:status = error }'       # 0

The structural operators go further and constrain the relationship. >> is “has a descendant matching”; > is “has a direct child matching”:

# Checkout traces where something anywhere below the entry point failed.
n '{ resource.service.name = "checkout-api" } >> { span:status = error }'   # 3

# Direct children only. The error span's parent is the checkout client span,
# so the descendant form matches and the child form is the narrower question.
n '{ span:name = "GET /payments/authorize" } > { span:status = error }'     # 3

This is the query an on-call engineer actually wants during the incident in the Scenario: show me entry-point traces whose failure is somewhere downstream. It cannot be expressed with a flat filter at all, which is why the trace store earns its place next to metrics and logs.

Task 7: Turn spans into numbers

An aggregate collapses the matching spans inside one trace into a single value, and then a comparator keeps or drops the trace:

# Traces with more than three spans: the four-span checkout traces.
n '{} | count() > 3'                                     # 19

# Traces with more than five spans: none in this corpus.
n '{} | count() > 5'                                     # 0

# Traces containing a payment span longer than a second.
n '{ resource.service.name = "payment-svc" } | max(duration) > 1s'   # 4

# The same four traces, asked as a trace-level question instead.
n '{ trace:duration > 1s }'                              # 4

The last two return the same set here and are not the same query. max(duration) reads the duration column of every matching span and reduces per trace; trace:duration is computed once per trace from the earliest start and the latest end. On this corpus the cost difference is invisible. On thirty days of production data it is the difference between a panel that renders and a panel that times out — and the two answers diverge as soon as a trace contains a long span that overlaps rather than extends the root.

Task 8: Prove one answer against the raw trace

A count is a claim. Take one trace ID from the descendant query and check it by hand, which is also how you learn the attribute names and types for the next query you write:

TID="$(tq '{ resource.service.name = "checkout-api" } >> { span:status = error }' 1 \
  | jq -r '.traces[0].traceID')"
echo "$TID"

curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | jq '[.. | objects | select(has("name") and has("spanId"))
         | {name, kind, status: (.status.code // 0),
            attrs: [(.attributes // [])[] | {(.key): .value}]}]'

Ask for JSON explicitly. The trace-by-ID endpoint returns protobuf unless the Accept header says otherwise, and a jq parse error on binary is the least informative failure in this lab.

Read three things out of that output. The root span’s status is 1, which is OK — the entry point genuinely succeeded. One span’s status is 2, which is ERROR, and it belongs to the payment service. And the http.response.status_code attribute is rendered as intValue, not stringValue, which is the evidence behind Trap 2: the type is visible right here, before you write the selector that depends on it.

Task 9: Measure what a query costs

The cost claims in the lessons are about column layout, and you can see the shape of them even on twenty-four traces. Time twenty runs of each and compare the medians:

timeit() {
  for i in $(seq 1 20); do
    curl -fsSG -o /dev/null -w '%{time_total}\n' "$TEMPO/api/search" \
      --data-urlencode "q=$1" \
      --data-urlencode "start=$START" --data-urlencode "end=$END" \
      --data-urlencode 'limit=100'
  done | sort -n | awk '{a[NR]=$1} END {printf "  p50=%.3fs  p95=%.3fs  n=%d\n",
        a[int(NR*0.5)], a[int(NR*0.95)], NR}'
}

echo "indexed intrinsic:"
timeit '{ span:status = error }'

echo "low-cardinality resource attribute:"
timeit '{ resource.service.name = "payment-svc" }'

echo "high-cardinality regex:"
timeit '{ span.session.id =~ ".*a.*" }'

At this scale every number is dominated by request overhead, and you should expect them to be close. That is worth seeing too: a benchmark on a corpus this small cannot demonstrate the difference, and reporting it as though it could would be dishonest. What the exercise establishes is the method — a fixed window, a fixed limit, twenty runs, a median rather than a single sample — which is the only way to compare two queries on a real cluster, where the difference is real and large.

session.id is in the corpus precisely because it is unique per trace. It is the shape of attribute — a request ID, a user ID, a full URL — that makes a filter expensive, because there is no useful dictionary for the column and every row has to be compared. The same attribute is what makes the query irreplaceable when you have the ID and need the trace, which is the trade-off rather than a mistake.

Task 10: What this lab deliberately does not show

Two parts of TraceQL are out of reach of a single monolithic Tempo with no extra components, and it is better to name them than to fake them:

TraceQL metrics — the rate(), count_over_time() and quantile_over_time() functions that turn a spanset into a time series. They need the metrics-generator with its local-blocks processor enabled, and on recent data they need it in the ingest path, not just at query time. Setting that up is the OTel-collector-pipeline lab’s territory.
Tenant isolation — every point in the query language where X-Scope-OrgID matters is invisible in a single-tenant lab, because auth_enabled is off and there is exactly one tenant. A wrong tenant header returns another team’s traces or none at all, and it is worth exercising on a cluster where the header does something.

Everything else in the TraceQL lessons is exercisable here, and the twenty-four traces are enough to be wrong against.

Validation

One transcript. Every line has an expected value; a mismatch is a real signal.

export TEMPO="${TEMPO:-http://127.0.0.1:3200}"
export START="$(date -d '-30 min' +%s)"
export END="$(date +%s)"

check() {  # query expected label
  got="$(curl -fsSG "$TEMPO/api/search" \
    --data-urlencode "q=$1" --data-urlencode "start=$START" \
    --data-urlencode "end=$END" --data-urlencode 'limit=100' \
    | jq '.traces | length')"
  if [ "$got" = "$2" ]; then
    printf '  ok    %-6s %s\n' "$got" "$3"
  else
    printf '  FAIL  got=%s want=%s  %s\n' "$got" "$2" "$3"
  fi
}

check '{}' 24 'whole corpus'
check '{ trace:rootService = "checkout-api" }' 19 'checkout entry point'
check '{ trace:rootService = "inventory-api" }' 5 'inventory entry point'
check '{ span:status = error }' 3 'failures anywhere'
check '{ trace:duration > 1s }' 4 'slow traces'
check '{ span.http.response.status_code = 500 }' 3 'http 500 as an integer'
check '{ span.http.response.status_code = "500" }' 0 'the quoted-integer trap'
check '{ span.service.name = "payment-svc" }' 0 'the wrong-scope trap'
check '{ resource.service.name =~ "payment" }' 0 'the unanchored-regex trap'
check '{ resource.service.name =~ ".*payment.*" }' 19 'anchored properly'
check '{} | count() > 3' 19 'four-span traces'
check '{ resource.service.name = "checkout-api" } >> { span:status = error }' 3 \
      'healthy root, failed descendant'

Twelve lines, twelve known answers. If one of them reports FAIL with got=0 across the board, re-run the window setup — that is Trap 4 rather than a bug in any individual query.

Expected Outcome

The bare {} matcher returns exactly 24 traces in the seeded window.
The four traps each return 0, and their corrected forms return 19, 3, 19 and 24 respectively.
{ span:status = error } returns 3, and a descendant query finds the same 3 from an entry point whose own status is OK.
A trace ID from that descendant query, fetched by ID, shows a root span with status 1 and a payment span with status 2.
{} | count() > 3 returns 19, separating the four-span checkout traces from the two-span inventory traces.
The timing helper reports three medians that are close together, and you can say why that is the honest result at this corpus size.

Troubleshooting

Every count is zero, including the bare matcher. In order: is the window right (echo $START $END, and are they unix seconds?); has a block been cut yet (the seed plus max_block_duration plus a margin); and is TEMPO pointing where you think. Confirm Tempo itself with curl "$TEMPO/ready".

The seed script prints something other than 200. A 400 is a rejected payload. Check date +%s%N first. A 000 means nothing is listening on 4318 — the receiver stanza binds 0.0.0.0 in this lab’s config for exactly that reason, because a container-local bind is unreachable from a published port.

/api/search returns HTTP 400 complaining about a parameter. The TraceQL query parameter is q. Older tag-based search used tags, and some tooling sends query. If your build rejects q, check the API reference for your tag rather than guessing between the three.

A count is off by a few. Re-running the seed adds another 24 traces rather than replacing them, and the window is thirty minutes wide. Either widen your expectations by the number of extra runs, or start clean: docker compose down -v && docker compose up -d.

jq: error ... not valid JSON on the by-ID lookup. The Accept: application/json header is missing and you are piping protobuf.

A structural query returns 0 where the flat one returns 3. Check the direction of the operator. A >> B reads “A has a descendant B”, so the left side is the ancestor. Reversing it asks a different and usually empty question.

Counts are right but .traces[0].traceID is null. The result field name differs between API versions. Run tq '{}' 1 | jq . and read the actual shape before adapting the filter.

Cleanup

cd "$HOME/traceql-lab"

docker compose down -v --remove-orphans
docker volume ls | grep tempo-data || echo "volume gone"

# Remove the image only if it was not already on the host before this lab.
# docker image rm "grafana/tempo:$TEMPO_TAG"

cd "$HOME"
rm -rf "$HOME/traceql-lab"

unset TEMPO OTLP_HTTP START END
ss -ltn 2>/dev/null | grep -E ':(3200|4318)' || echo "both lab ports are free again"

If you pointed at an existing Tempo instead of starting one, the seeded traces are still in it. They age out at that instance’s block_retention, and until then they are findable — and removable, if the instance is yours — with { resource.deployment.environment = "lab" }.

What You Learned

A query result you cannot check is not evidence. The whole design of this lab is that twenty-four traces in four known shapes turn every count into a pass or a fail. The same trick works in production: seed a canary trace with a service name nobody else uses, and your dashboard queries become testable.
An empty result is four different bugs wearing the same face. Wrong scope, quoted integer, unanchored regex, wrong window. You produced all four. The bare {} matcher is the one diagnostic that separates the last from the first three, which is why it belongs first in the order.
The failure is usually not at the entry point. The three failed traces return 200 at the front door. A filter on the entry service’s own status finds none of them; { A } >> { B } finds all three. Systems with retries and fallbacks produce this shape by default, not by accident.
Braces are scoping, not punctuation. { A && B } asks one span to satisfy both conditions; { A } && { B } asks the trace to contain both, possibly on different spans. Reading a saved query wrongly on this point is how two people argue about a dashboard that is doing exactly what it says.
Intrinsics carry no type ambiguity and survive schema churn. span:status = error cannot be broken by quoting, and it keeps working when the HTTP attribute names change under an SDK upgrade. Reach for the intrinsic first and the attribute only when the question needs it.
A benchmark that cannot show the difference should say so. The timing exercise establishes the method — fixed window, fixed limit, twenty runs, a median — and reports honestly that at this scale the medians converge. That is a more useful result than three numbers presented as a finding.

Taking this to a real cluster

Three things change when the corpus is real. The window stops being free. A search over thirty days scans every block in the window, so a max_query_length on the querier is not a nicety — without one, a single user with a wide time picker can saturate the queriers for everyone. Set it before you need it.

Every saved query needs a limit. A TraceQL search without one returns as many matches as the engine will hold, and a Grafana panel that renders all of them is a browser tab that stops responding during an incident. Twenty is a reasonable panel default; ten for exploration.

Define the selector once. The service-name filter you write today will be copied into a dozen panels, several alert annotations and at least one runbook, and the copies will drift — some on resource.service.name, some on span.service.name, all of them returning zero in the same incident and none of them obviously wrong. Put it in a dashboard variable or a documented query and reference it, exactly as you would for a recording rule.

Finally, keep a canary. One trace per minute from a fixed synthetic service, on a service name reserved for the purpose, gives you a query with a known answer at any moment — which turns “is the trace store healthy?” into a check instead of a debate.

Objectives

Prerequisites