Tempo Upgrade Basics

A team runs Tempo 2.3 in production. The team schedules an upgrade to 2.6 on a Wednesday afternoon. The chart bumps the image, the ingester pods restart one at a time, and the distributor ring rebalances. The team watches the ingester metric tempo_ingester_local_blocks fall, then rise again as each pod replays its WAL. Thirty minutes in, the search panel returns empty results for any tag-based query. Trace-by-ID still works. The team investigates and finds the new compactor is dropping blocks because the search backend has been split out into a separate binary that the chart forgot to enable. The fix is to enable compactor.compaction.block_search_encoding: v2 and compactor.compaction.search_relevant_tag_values, wait for the compactor to run one cycle, and document the change.

Tempo upgrades do not fail in exotic ways. They fail because three decisions were either made in the wrong order or not made at all. This lesson is the discipline that prevents that failure shape.

What it is

A Tempo upgrade is the process of moving a running Tempo cluster from one version to a newer version while preserving the ability to read all traces written by the previous version. The upgrade is composed of three independent concerns:

• Storage format compatibility. The block format the ingester writes and the compactor merges. Tempo has shipped v1 (proto columnar), v2 (indexed columnar), and vParquet. Each upgrade can add a new format while keeping the old ones readable. The compactor rewrites old blocks as a background job.
• Ring membership compatibility. The hash ring for the ingester and the compactor. A mixed-version ring is supported only within the version matrix published in the release notes. The default is fully stateless for the duration of a minor-version cutover and stateful only within a single minor step.
• Receiver surface compatibility. The set of receivers enabled on the distributor (OTLP gRPC, OTLP HTTP, Jaeger, Zipkin, Kafka) and any protocols removed or renamed. A receiver that disappears in the target version will silently stop accepting traffic from the clients that still depend on it.

Each is a separate decision. A configuration-only upgrade can be applied at the rolling restart. A storage upgrade requires that the compactor finishes its migration cycle before the WAL is reaped. A receiver removal requires that the affected client fleets are migrated to OTLP before the upgrade.

Why a sysadmin cares

The Tempo release cadence is roughly monthly. Standing still is not an option: security patches and ingester fixes land in the supported series, and the gap between the running version and the latest stable grows every cycle. The cluster that skips four minor versions pays the surcharge in a single larger upgrade window, during which the diff between the running config and the target config exceeds the team’s capacity to review it.

Three risks dominate the upgrade conversation:

1. Read regression. A version bump that changes the default block format or the search backend produces a cluster that writes the new format but reads the old format inconsistently. Queries that span the cutover return empty or partial results.
2. Ring churn regression. A version bump that rewires the ingester hash ring membership (for example, by changing the lifecycler.ring.replication_factor default or the heartbeat timeout) produces brief 503 windows during the roll. The window extends when the chart or the values file does not pin the new defaults.
3. Receiver removed regression. A version bump that removes a receiver (Jaeger Thrift in Tempo 2.7, OpenCensus in Tempo 2.0) silently breaks every client still pushing through it. Trace ingest drops to zero for those clients before they notice.

The cost of a botched upgrade is not the alert that fires. It is the days of post-incident analysis while the team reconstructs what the cluster was doing before the restart.

How it works

The upgrade is a sequence of independent decisions. The diagram below is the cutover order that has shipped safely in most Tempo deployments; each step has its own validation.

        Pre-upgrade
            |
            v
   +-------------------+   Capture baseline
   | Read release      |   (WAL size, block count,
   | notes for target  |    ring members, query latency)
   +-------------------+
            |
            v
   +-------------------+   helm diff
   | Diff Helm values  |   against running values
   | against target    |
   +-------------------+
            |
            v
   +-------------------+   Roll stateless first
   | Distributor       |   (no data state, drain
   | Querier           |    safe)
   | Query-frontend    |
   +-------------------+
            |
            v
   +-------------------+   Roll stateful one at a time
   | Ingester          |   WAL replays on each restart,
   |                    |   ring rebalances
   +-------------------+
            |
            v
   +-------------------+   Roll compactor singleton
   | Compactor          |   (or sharded if enabled)
   +-------------------+
            |
            v
   +-------------------+   Run one full cycle
   | Compactor          |   so old blocks are
   | completes cycle    |   rewritten to new format
   +-------------------+
            |
            v
        Soak window

The right cutover order is “stateless first, then stateful, then compactor, then a soak”. Inverting any step produces a cluster that is writing the new format before the read path is ready to serve it.

Under the hood

How to configure it

The upgrade is a values diff plus a measured rollout. The minimal tempo section that changes between versions is below; the rest of the config is unchanged.

# tempo.yaml -- values that move between minor versions
server:
  http_listen_port: 3200
  grpc_listen_port: 9095

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: '0.0.0.0:4317' }
        http: { endpoint: '0.0.0.0:4318' }
    # jaeger: removed in 2.7. Commented out to confirm intent.
    zipkin:
      endpoint: '0.0.0.0:9411'

ingester:
  trace_idle_period: 10s
  max_block_duration: 30m
  lifecycler:
    ring:
      kvstore: { store: memberlist }
      replication_factor: 3
      heartbeat_timeout: 5m
    join_after: 10s
    observe_period: 10s

compactor:
  compaction:
    block_retention: 48h
    compaction_window: 1h
    block_search_encoding: v2

storage:
  trace:
    backend: s3
    s3:
      bucket_name: tempo-traces-prod
      region: eu-west-1
    wal:
      path: /var/tempo/wal

Three keys to call out:

• distributor.receivers.jaeger was removed in Tempo 2.7. The comment above is the operational discipline — keep the removal visible in the values diff rather than letting the chart delete the key in silence.
• compactor.compaction.block_search_encoding: v2 was added to tell the compactor to rewrite older v1 blocks in the new indexed format. Without it the compactor merges but does not upgrade.
• ingester.lifecycler.ring.heartbeat_timeout: 5m should be pinned explicitly. The default was tightened in a recent minor release and is a common silent regression.

How to validate it

Severity: READ-ONLY.

1. Capture pre-upgrade baselines — the numbers you compare against after the cutover:

echo "ingester_local_blocks (pre)"; \
  curl -s http://tempo-ingester:3200/metrics \
    | awk '/^tempo_ingester_local_blocks{/ {print $1, $2}'
echo "compactor_blocks_compacted_total (pre)"; \
  curl -s http://tempo-compactor:3200/metrics \
    | awk '/^tempo_compactor_blocks_compacted_total / {print $2}'
echo "querier_query_seconds p99 (pre)"; \
  curl -s http://tempo-querier:3200/metrics \
    | awk '/^tempo_querier_query_seconds_bucket{le="+Inf"}/ {print $2}'

2. Read the rolling-restart order in the chart and verify the StatefulSet was selected for ingester and compactor:

kubectl -n observability get statefulset \
  -l app.kubernetes.io/name=tempo,app.kubernetes.io/component \
  --show-labels
# COMPONENT matches the Helm-managed target for ingester / compactor.

3. Confirm a synthetic trace round-trips end to end (writing via the configured receiver and querying by ID):

TRACE=$(uuidgen)
curl -sG http://tempo-distributor:4318/v1/traces \
  -H 'content-type: application/x-protobuf' \
  --data-binary @<(
    otel-cli span export --service synth --name probe --trace-id "$TRACE"
  )
sleep 5
curl -sG http://tempo-querier:3200/api/traces/$TRACE | jq '.batches | length'
# 1

4. After the cutover completes, confirm the new version is actually running:

kubectl -n observability get pods -l app.kubernetes.io/name=tempo \
  -o jsonpath='{range .items[*]}{.metadata.labels.tempo\.g\.rafana\.co/version}{"\n"}{end}' \
  | sort -u
# v2.6.0

5. Confirm the compactor is making progress on the bucket:

curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_blocks_compacted_total / {print "blocks:", $2}'
curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_compaction_errors_total / {print "errors:", $2}'
# blocks: 4231
# errors: 0

How it can fail

Six shapes appear repeatedly in production Tempo upgrades:

1. Block format silent mismatch. The ingester is rolled but the compactor is still on the old version with old defaults. Symptom: tempo_compactor_block_format_upgrades_total is flat while the bucket still holds v1 blocks. Queries return results only when the requested trace ID is in a v2 block.
2. Mixed ring membership. A new ingester pod cannot reach the old ingester pods over memberlist. Symptom: tempo_ingester_lifecycler_ring_inconsistencies_total is rising and the distributor returns no healthy ingesters.
3. Receiver silently removed. Tempo 2.7 dropped Jaeger receivers. A pod stays up but accepts no Jaeger traffic. Symptom: tempo_distributor_spans_received_total drops to zero for the Jaeger protocol while metrics for OTLP keep moving.
4. WAL not drained before reaping. A pod rolls before its WAL is fully replayed. Symptom: missing traces for the window the pod was down. The metric tempo_ingester_wal_replay_duration_seconds shows that the replay was truncated.
5. Search backend misconfigured. The vParquet migration is partial. Symptom: trace-by-ID queries return results, tag-based searches return empty. The metric tempo_query_frontend_queries_with_errors_total rises.
6. Compactor left at the old version. The compactor is a singleton and is rolled last. Symptom: bucket grows because the new ingester format is incompatible with the old compactor, and tempo_compactor_blocks_compacted_total is flat.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Did the binary actually change? kubectl get pods and the container image digest. A Helm re-install that resolves to the same tag restarts pods without upgrading.
2. Is every role ready? /ready per role. A 503 is a wrong target flag or a missing config map.
3. Is the ring healthy? /ingester/ring and /compactor/ring. A reduced member list means a pod is still in JOINING.
4. Are spans arriving? tempo_distributor_spans_received_total and tempo_ingester_spans_received_total. A flat distributor counter is a client-side problem.
5. Are blocks flushing? tempo_ingester_local_blocks. A flat counter for the lifecycle of a pod, then a sharp drop, is the healthy replay pattern.
6. Is the compactor keeping up? tempo_compactor_blocks_compacted_total and the compactor log file. Stale compaction logs point at storage credential rotation or a broken bucket name.

Security implications

Three surfaces change between releases:

• Receiver ports change. A receiver removed in the new version leaves a stale Service and NetworkPolicy that no longer match the binary. The Service continues to route traffic to a port that nothing listens on; client mTLS errors pile up in the collector.
• Bucket credentials. The ingester and compactor both write to the same bucket. After an upgrade, the new binary expects the credentials under the env-var name it uses; old secrets named for the previous binary are silently ignored.
• mTLS between roles. Tempo 2.x tightened the default authentication between the querier and the query-frontend. A cluster that enabled client: true for the querier must renew the cert before the binary on the other side enforces the check.

Performance implications

The cost profile differs by version:

• Block size. vParquet blocks compress better than v2, reducing storage cost and improving query fan-out latency. The trade-off is a slower compactor cycle because the rewrite is I/O-bound.
• WAL replay time. Larger max_block_duration reduces the flush rate but lengthens the WAL replay on pod restart. A pod that takes longer to replay is a pod that returns 503 on the next trace.
• Query path. TraceQL aggregations (count(), avg()) became cheaper across minor releases because the search backend indexed the relevant tags. The win disappears if the compactor never runs to rewrite the old blocks.

Production guidance

• Read the release notes section by section: breaking changes, deprecations, configuration renames, operational notes. The diff between running and target is more important than the list of changes.
• Capture three baselines before the upgrade: WAL size, block count, query latency p99.
• Roll stateless first (distributor, querier, query-frontend), then stateful (ingester), then singleton (compactor).
• Keep at least one ingester on the previous version for one full max_block_duration window so the ring does not empty.
• Do not delete the previous values file from git until the compactor has run one full cycle and the rollback path has been re-tested against the new values.

Verification

You should now be able to answer:

• What are the three categories of Tempo upgrade risk, and which one produces a read regression?
• Which role do you roll first in a rolling upgrade, and why?
• What baseline do you capture before the cutover?
• What is the first diagnostic that proves the new binary is actually running and not just the old binary restarted by a Helm re-install?
• How do you confirm the compactor has finished migrating the bucket to the new block format?

Quiz