Tempo Validation

A team deploys a new Tempo config to staging. The rollout appears successful; the pods are running; the service is reachable. Two days later, the on-call engineer investigates a slowdown and opens Tempo — empty. The team discovers that distributor.receivers.otlp.protocols.grpc was missing the endpoint key, which Tempo silently accepted as a default of “do not bind”. The pod started, the readiness probe returned 200, and the OTLP exporter was sending to a closed port. The config validator would have caught it before deploy. There was no validator in the pipeline.

Validation is the discipline that catches this kind of silent misconfiguration before it reaches production.

What it is

Tempo validation is the set of checks that confirm a Tempo deployment is configured correctly and behaving correctly before and after a change. There are three layers:

1. Static config validation. tempo-cli validate-config path/to/tempo.yaml parses the YAML, applies Tempo’s schema, and reports errors with file and line numbers.
2. Runtime readiness. The /ready HTTP endpoint on each role returns 200 only when that role is registered and healthy.
3. Synthetic trace. Push a known trace via OTLP, then read it back via the HTTP API. This proves the path (receiver -> distributor -> ingester -> storage -> querier) is end-to-end functional.

A deployment that passes all three is verified. A deployment that passes only the first is config-valid but not yet behaviour-valid.

Why a sysadmin cares

Tempo has a permissive parser. Unknown keys are ignored, not rejected. A typo in endpoint does not fail the start; the role simply does not bind the port. The validator is the backstop.

Readiness probes are reliable for “the process is alive”. They are not reliable for “the receiver port is bound” — the role can be registered without the receiver stanza taking effect. The synthetic trace is the proof.

How it works

                  tempo-cli               curl /ready
                  ---------               ----------
   tempo.yaml  --+      +- pass / fail    pod  ->  200 / 404
                  \    /                    \
                   \  /                      \
                    \/                        \
                config validation              per-role readiness
                                                \
                                                 \
                                  synthetic trace \
                                  -----------------
   otel-cli span  --+      +--- tempo HTTP API   /
                    \    /        \            /
                     \  /          \          /
                      \/            \        /
                write a trace        read it back

The three layers are independent. Each catches a class of failure the others do not.

• tempo-cli catches schema and parse errors.
• /ready catches process-level failures (crash, hang, missing role registration).
• The synthetic trace catches data-path failures (port not bound, bucket unreachable, tenant mis-routing).

Under the hood

How to validate it

Layer 1: static config

tempo-cli validate-config /etc/tempo/tempo.yaml

Real output (success):

tempo-cli version: 1.5.0
config is valid: /etc/tempo/tempo.yaml
exit 0

Real output (failure — missing receiver port):

level=error msg="error parsing config" 
  error="yaml: line 42: did not find expected key"
level=error msg="error parsing config"
  error="distributor.receivers.otlp.protocols.grpc: endpoint required"
exit 1

Severity: READ-ONLY. The validator never modifies the file.

Layer 2: per-role readiness

for role in distributor ingester querier compactor query-frontend metrics-generator; do
  echo -n "$role: "
  curl -s -o /dev/null -w "%{http_code}\n" \
    http://tempo-${role}:3200/ready
done

Real output (success):

distributor: 200
ingester: 200
querier: 200
compactor: 200
query-frontend: 200
metrics-generator: 200

A 404 on any line means the role is not registered in that pod. A 000 means the pod is unreachable.

Layer 3: synthetic trace

# Push a synthetic span
TRACE_ID=$(otel-cli span export \
  --endpoint tempo:4317 \
  --service-name validate-tempo \
  --name "smoke-test" \
  --kind server | awk -F= '/trace_id/ {print $2}')

echo "Pushed trace: $TRACE_ID"

# Wait for the ingester to flush (default 15 minutes; force
# a flush in test by lowering flush_to_storage, or wait).
sleep 30

# Read it back
curl -sG "http://tempo:3200/api/search" \
  --data-urlencode 'query={ resource.service.name = "validate-tempo" }' \
  --data-urlencode 'limit=5' | jq .

Real output:

$ curl -sG "http://tempo:3200/api/search" \
    --data-urlencode 'query={ resource.service.name = "validate-tempo" }' \
    --data-urlencode 'limit=5' | jq '.traces | length'
1

A zero means the trace was not flushed yet, or the search returned no results. A traceID field with the value of $TRACE_ID confirms the round-trip.

Compose a smoke test

The three layers combined into a single bash script:

#!/usr/bin/env bash
set -euo pipefail

CONFIG=${1:-/etc/tempo/tempo.yaml}
ENDPOINT=${TEMPO_ENDPOINT:-http://tempo:3200}
OTLP=${OTLP_ENDPOINT:-tempo:4317}

echo "== 1. Static config =="
tempo-cli validate-config "$CONFIG"

echo "== 2. Readiness =="
for role in distributor ingester querier compactor; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "$ENDPOINT/ready")
  echo "  /ready on $ENDPOINT -> $code"
  [[ "$code" == "200" ]] || { echo "role not ready"; exit 1; }
done

echo "== 3. Synthetic trace =="
TRACE_ID=$(otel-cli span export \
  --endpoint "$OTLP" \
  --service-name tempo-smoke \
  --name "smoke-test" \
  --kind server | awk -F= '/trace_id/ {print $2}')

echo "  pushed trace_id=$TRACE_ID"
sleep 30

found=$(curl -sG "$ENDPOINT/api/search" \
  --data-urlencode "query={ resource.service.name = \"tempo-smoke\" }" \
  --data-urlencode 'limit=5' \
  | jq --arg t "$TRACE_ID" '.traces[] | select(.traceID == $t) | .traceID')

[[ -n "$found" ]] || { echo "synthetic trace not found"; exit 1; }
echo "smoke test PASSED"

Severity: READ-ONLY for layers 1 and 2. Layer 3 pushes one synthetic trace; the data is tiny but should be allowed by the retention policy.

How it can fail

1. tempo-cli not installed on the deploy host. The pipeline falls back to “apply and see what happens”. The misconfiguration reaches production. Symptom: 503 from the receiver port; no spans ingested.

2. tempo-cli reports success but the runtime fails. The validator checked structure, not runtime. A misconfigured bucket or wrong tenant header passes validation. Symptom: pods start; /ready returns 200; writes fail silently.

3. Readiness probe returns 200 but the OTLP port is not bound. The receiver stanza had a typo that Tempo silently ignored at startup. Symptom: ss -tlnp | grep 4317 returns empty; client connections time out.

4. Synthetic trace never flushed to storage. The default flush_to_storage is 15 minutes; the smoke test waits 30 seconds; the search returns empty. Symptom: false negative on the smoke test; team believes Tempo is broken when it is merely waiting for the flush.

5. /ready returns 200 on a querier-only deployment that has no distributor. A misconfigured microservices deployment puts only the querier in front; the OTLP endpoint is unreachable. Symptom: smoke test fails at layer 3, but layer 2 reports success.

6. Smoke test runs against production. A misconfigured CI pipeline points at the production Tempo. The synthetic trace lands in the production bucket; retention deletes it on schedule, but the trace ID is recorded in audit logs. Symptom: noise in audit; potential disclosure if the service name embeds sensitive data.

How to troubleshoot it

Order of diagnostics, cheapest first:

1. Does tempo-cli validate-config pass? If not, fix the YAML and try again. There is no point in checking readiness on a config that does not parse.
2. Is each role registered? curl /ready on each role. Resolve 404s before running the synthetic trace.
3. Are the receivers bound? ss -tlnp | grep 4317 on the distributor host. An empty list means the receiver stanza did not bind a port.
4. Is the synthetic trace landing? Look at the distributor’s tempo_distributor_spans_received_total counter. A zero delta means the export did not reach the receiver.

Security implications

• Smoke test traffic in production. A synthetic trace pushed to production contains a service name; if the service name embeds sensitive data (a tenant ID, an environment name), it lands in production storage and in audit logs. Use a fixed service name for smoke tests.
• tempo-cli in CI. The CI host needs read access to the Tempo YAML. If the YAML contains secrets (it should not — prefer environment variables), CI extracts them.
• Readiness probe as an attack surface. /ready returns “ready” with no authentication. In a hostile network, an attacker who can reach the Tempo HTTP port can probe readiness to enumerate roles. Bind to a private interface.

Performance implications

• Validator cost. tempo-cli validate-config parses the YAML in milliseconds. Run on every CI build.
• Readiness probe cost. A 1-Hz probe per pod is the usual Kubernetes default. The /ready handler is a static “200 OK” with no I/O; the cost is negligible.
• Synthetic trace cost. One trace per smoke test run. At default settings the trace is flushed in 15 minutes. The storage cost is rounding error; the operational signal is worth the bytes.

Production guidance

• Run tempo-cli validate-config in CI on every pull request that touches tempo.yaml.
• Run the readiness probe and synthetic trace as part of the post-deploy hook in your CD pipeline.
• Use a fixed service name (tempo-smoke, tempo-validate) for the synthetic trace so it can be filtered out of per-tenant searches.
• For high-stakes changes (a new receiver, a new bucket), require the full smoke test to pass before merge.

Verification

You should now be able to answer:

• Which three layers should a Tempo validation pipeline cover?
• What does tempo-cli validate-config actually check?
• Why is a 200 on /ready not sufficient evidence that the OTLP receiver is bound?
• What is the default wait between an OTLP push and a trace becoming searchable?
• Why should the synthetic trace use a fixed service name?

Quiz