Tempo Upgrade Validation

A team has just rolled Tempo from 2.5 to 2.6. The team runs a smoke test: pushes a synthetic trace through the OTLP receiver, queries it back by ID, sees the result. The smoke test passes. The team marks the upgrade complete. The on-call engineer walks away. Two hours later, an operator opens a Grafana dashboard that pivots into traces filtered by the resource.k8s.namespace.name attribute. The dashboard is empty. The team investigates. The smoke test only validated the service.name filter; the compactor was never configured to index the new attribute; the dashboard references an attribute that lives only on the index of the new blocks, not on the older blocks the bucket still holds.

Validation is not a smoke test. A smoke test confirms the thing that was just deployed. Validation confirms the thing that was deployed against the thing that was already there.

What it is

Validation is the post-upgrade process of confirming that the running cluster is serving the same data as the cluster before the upgrade. Validation is a checklist, not a single test. The checklist has four parts:

1. Binary validation. Every pod is running the target version. The binary actually changed.
2. Config validation. The pod loaded the new config. The renamed keys took effect. The removed keys are absent from the pod log.
3. Storage validation. The read path is serving from the bucket. The compactor is making progress. The block format distribution is moving in the expected direction.
4. End-to-end validation. A synthetic trace is written every minute, queried back, and the result matches. The check runs continuously and alerts on failure.

The four parts run in order. A binary validation failure is a hard block. A config validation failure is a hard block. A storage validation failure is a hard block. An end-to-end validation failure is a soft block (the upgrade may be complete; the validation may be wrong).

The synthetic trace is the foundation of the end-to-end validation. The trace is emitted by otel-cli (or a custom agent) and queried back through the same Tempo API Grafana uses. The agent reports four metrics:

• tempo_canary_last_success_timestamp_seconds — the timestamp of the last successful round trip.
• tempo_canary_writes_total — the count of successful writes.
• tempo_canary_reads_total — the count of successful reads.
• tempo_canary_query_duration_seconds — the histogram of query latency.

The first metric is the proof the cluster is serving the new data. The fourth metric is the proof the cluster is serving at the expected latency.

Why a sysadmin cares

Validation is the difference between an upgrade that ships and an upgrade that wakes the on-call engineer at 02:00. Three production scenarios apply:

1. The config diff was not validated. A renamed key was silently ignored by the binary. The cluster reads from the chart default. The team discovers the regression when a user reports an empty query.
2. The block format migration was not validated. The compactor’s block_search_encoding was not updated. The bucket still holds the old format. The index of the new attributes does not exist on the old blocks. The team discovers the regression when a dashboard pivots on the new attribute.
3. The synthetic trace was not continuous. The validation run only on Monday morning. The first signal of a regression is the user report. The team has no way to distinguish a regression from a transient.

The cost of a missing validation is the days of post-incident analysis while the team reconstructs what the cluster was doing before the upgrade. The cost of a present validation is the five minutes of checklist execution that confirms the upgrade is complete.

How it works

The validation is a four-part checklist with a synthetic trace at the centre. The shape is the same every upgrade; the names change.

   Binary validation
        |
        v
   Pod image tag == target
        |
        v
   Config validation
        |
        v
   "unexpected config key" log scan
        |
        v
   Per-component metrics emit
        |
        v
   Storage validation
        |
        v
   Block format distribution moves
        |
        v
   Compactor cycle makes progress
        |
        v
   End-to-end validation
        |
        v
   Synthetic trace round-trip:
       write succeeded
       query returned the trace
       latency below baseline

The synthetic trace is the cornerstone because it exercises the receiver, the distributor, the ingester, the compactor cycle, the query-frontend, and the querier in one round trip.

Under the hood

How to configure it

The synthetic trace is a deployment of otel-cli running on a cron schedule. The validation is a small script that hits the Tempo API and confirms the trace.

# tempo-canary CronJob -- one push and one query every minute
apiVersion: batch/v1
kind: CronJob
metadata:
  name: tempo-canary
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: emit
              image: otel-dummy-1.0
              args:
                - --endpoint=tempo-distributor:4317
                - --protocol=otlp/grpc
                - --service=synth
                - --name=probe
                - --trace-id=$(TRACE_ID)
                - --attribute=host=$(POD_NAME)
                - --attribute=run=$(COUNTER)
            - name: query
              image: curlimages/curl
              args:
                - -sG
                - http://tempo-querier:3200/api/search
                - --data-urlencode=q={ service.name = "synth" }
                - --data-urlencode=limit=1

The Tempo side does not need to know the canary is running. The validation runs against the public Tempo API the Grafana data source uses.

How to validate it

Severity: READ-ONLY.

1. Confirm every pod is running the target version:

kubectl -n observability get pods -l app.kubernetes.io/name=tempo \
  -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.metadata.labels.tempo\.g\.rafana\.co/version}{"\n"}{end}' \
  | sort -u
# tempo-compactor-0: v2.6.0
# tempo-distributor-abcde: v2.6.0
# tempo-ingester-0: v2.6.0

Every line should report the target version. A line that reports the previous version is a pod that has not yet been recreated.

2. Confirm every role is ready:

for role in distributor ingester querier query-frontend compactor metrics-generator; do
  echo -n "$role: "
  curl -s "http://tempo-${role}.observability.svc:3200/ready"
  echo
done
# distributor: ready
# ingester: ready
# querier: ready
# query-frontend: ready
# compactor: ready
# metrics-generator: ready

3. Confirm the pod logs are free of unexpected config key warnings. The warnings list the renamed or removed keys that are still in the values file:

for role in distributor ingester querier query-frontend compactor metrics-generator; do
  echo -n "$role: "
  kubectl -n observability logs deploy/tempo-${role} --tail=50 \
    | grep -E "unexpected config key" \
    | wc -l
done
# distributor: 0
# ingester: 0
# querier: 0
# query-frontend: 0
# compactor: 0
# metrics-generator: 0

4. Confirm the synthetic trace round-trips:

TRACE=$(uuidgen)
otel-cli span export --service synth --name probe --trace-id "$TRACE" \
  --endpoint tempo-distributor.observability.svc:4317 \
  --protocol otlp/grpc
sleep 10
RESULT=$(curl -sG http://tempo-querier:3200/api/search \
  --data-urlencode "q={ traceID = \"$TRACE\" }" \
  | jq '.traces | length')
echo "round-trip result: $RESULT"
# round-trip result: 1

5. Confirm the compactor is making progress. Compare the metric to the pre-upgrade value:

curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_blocks_compacted_total / {print "compacted:", $2}'
curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_compaction_errors_total / {print "errors:", $2}'
# compacted: 4231
# errors: 0

A counter that is rising and an errors counter that is flat are the proof the migration is healthy.

6. Confirm the bucket block format is moving:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix tempo/ \
  --output json \
  --query "Contents[].Key" \
  | jq -r '.[]' \
  | head -n 200 \
  | while read k; do
      aws s3 cp "s3://tempo-traces-prod/$k/meta.json" - 2>/dev/null \
        | jq -r '.data_encoding // .version // "unknown"'
    done \
  | sort | uniq -c
# 482 v2
# 198 vParquet

The vParquet count rising is the proof the compactor is producing the new format. The v2 count steady or falling means the migration is not regressing.

7. Confirm the read path is responsive:

curl -w "%{time_total}\n" -o /dev/null -s \
  http://tempo-querier:3200/api/search \
  -G --data-urlencode 'q={ resource.service.name = "checkout" }' \
  --data-urlencode 'limit=5'
# 0.421

How it can fail

Six shapes appear in production post-upgrade validations:

1. Binary mismatch. A pod that was supposed to roll reports the previous version. Symptom: a label mismatch across the deployment; the rollout controller has stalled.
2. Receiver not bound. A removed or renamed receiver leaves the port unbound. Symptom: the pod log shows unknown receiver or no transport bound.
3. Compactor lag. The migration of the old format is partially drained. Symptom: tempo_compactor_blocks_compacted_total is rising but the bucket block count is not dropping.
4. Cache TTL too long. A Grafana panel that returned empty before the cutover caches the empty result. Symptom: the dashboard panel shows empty for up to the cache TTL.
5. WAL replay incomplete. The first ingester pod to roll has not finished replaying its WAL. Symptom: missing traces for the window the pod was down.
6. Synthetic trace path wrong. The canary pushes to the wrong receiver (for example localhost instead of the distributor). Symptom: the synthetic trace never appears; the round-trip metric shows zero.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Did the binary actually change? Pod image tags and label values. A chart re-install that resolves to the same image tag restarts pods without upgrading.
2. Did the config load? Pod log scan for unexpected config key. A warning list of three keys is three keys the chart defaults are about to apply.
3. Is the synthetic trace landing? otel-cli span export directly to the distributor. A 400 error is a wrong port or a wrong protocol.
4. Is the read path responsive? A direct curl against the querier. A 500 or timeout is a misconfigured service.
5. Is the compactor making progress? The metric tempo_compactor_blocks_compacted_total should be monotonically increasing during the migration window.

Security implications

The validation exercises the same surfaces that the production fleet uses. There is no separate surface.

• Synthetic trace PII. The canary span must use a service name and a set of attributes that do not collide with a production service. A service name like synth-canary is recommended.
• Receiver credentials. The canary push uses the OTLP receiver without authentication (in development). Production clusters that wrap OTLP behind mTLS or an API key must configure the canary the same way.
• Bucket access. The bucket-format listing requires s3:GetObject and s3:ListBucket. The scope of the IAM credentials used by the validation script must match the production reader scope.

Performance implications

The validation script produces a load that is small but not zero:

• Synthetic trace. One OTLP push and one search every minute. The push is well within the receiver rate limit.
• Block listing. A list-objects-v2 of a few hundred keys is a small fraction of one cycle of the production bucket scane.
• Compactor cycle. The validation does not run a compactor cycle. It reads the compactor metric the production compactor emits. The validation script does not perturb the compactor workload.

Production guidance

• Treat the validation script as a small SRE task: it has an owner, a runbook, and a primary and secondary on-call contact.
• Run the validation script in CI; the validation that ships with the upgrade is the validation that catches the regression.
• Version-pin the synthetic trace agent. A new agent version may push different attribute sets; the validation expectations need updating.
• Document the post-upgrade baseline in the upgrade runbook. A new validation run is the proof the cutover is complete; a missing baseline is the reason the team cannot interpret the result.

Verification

You should now be able to answer:

• What is the four-part structure of a Tempo upgrade validation, and which part is the soft block?
• What is the cornerstone of the end-to-end validation, and which four metrics does it expose?
• Which three metrics prove the compactor is making progress on the bucket?
• What command confirms every pod is running the target version, not the previous version?
• What does a non-zero unexpected config key warning count in the pod log prove?

Quiz