Block Format Compatibility

A team runs Tempo 2.3 with the v2 block format. The team plans an upgrade to the next minor release. The release notes announce vParquet as the recommended default and note that v2 blocks remain readable but no longer mergeable into newer blocks without a compactor rebuild. The team reads the note, rolls the chart, and walks away. Two weeks later the bucket has five million blocks because the compactor was never reconfigured to rewrite the v2 blocks. Query latency is climbing. The team investigates, finds that compactor.compaction.block_search_encoding was not set, and runs a manual compaction to drain the backlog.

Block format migrations are not detected by the upgrade command. They are detected by the compactor cycle. This lesson covers the lineage of formats, the migration mechanics, and the metric that proves the migration is complete.

What it is

A Tempo block is a self-contained directory in object storage that holds the spans for one to thirty minutes of ingest time. Every block carries a format header that identifies how the data is encoded. Tempo has shipped three generations of format over its lifetime:

• v1. Columnar storage based on the proto marshaller. Default through Tempo 1.x and the early 2.x series.
• v2. Indexed columnar storage with a separate index file per block. Introduced as the default in Tempo 2.0 to make tag-based searches cheaper.
• vParquet. Apache Parquet columnar storage, with the tag index embedded in the file footer. Introduced as the recommended default in the Tempo 2.5 / 2.6 release window and tightened as mandatory for new ingester outputs.

The three formats are not equivalent. They differ in on-disk size, in query fan-out cost, and in the work the compactor must do when it merges two blocks of different versions. The line that matters in production is that a new ingester can read every block format the old ingester produced, but the compactor must be reconfigured to rewrite the older formats.

The migration is therefore a compactor job, not an ingester change. The ingester change is the trigger; the compactor job is the work.

Why a sysadmin cares

Three production scenarios apply:

1. Bucket growth without query growth. A bucket that the ingester writes to in vParquet but the compactor leaves at v2 has thousands of small v2 blocks and the larger vParquet blocks the ingester just flushed. The ratio of block count to trace count creeps up. The querier pays the cost on every tag-based query because the index file is older and smaller.
2. Compactor error spam. A compactor that does not know the vParquet format leaves v2 blocks unmerged. The metric tempo_compactor_compaction_errors_total rises and the compactor log file shows unknown block version for the older entries.
3. Storage bill growth. vParquet compresses better than v2 by a noticeable margin. A cluster that never migrates pays the old per-byte cost permanently.

The cost is not the upgrade itself. The cost is the bucket that drifts out of the expected shape while nobody watches the compactor metric.

How it works

A block format migration has three observable phases:

   Pre-upgrade bucket
        |
        |  v1 blocks    v2 blocks    no vParquet
        v
   +-------------------+
   | Format headers in |  tempo-cli or `s3 ls`
   | block listing     |  --output json
   +-------------------+
        |
        v
   +-------------------+  ingester rolls,
   | New ingester      |  starts writing vParquet
   | writes vParquet   |  alongside older reads
   +-------------------+
        |
        v
   +-------------------+  compactor rolling,
   | Compactor learns   |  rewrites old blocks
   | new format        |  into the new one
   +-------------------+
        |
        v
   +-------------------+  cleanup pass deletes
   | Bucket holds only |  empty parents once
   | vParquet blocks   |  the merge completes
   +-------------------+

The migration is safe at every step. The new ingester reads every old block; the compactor merges pairs of blocks when both participants are within the compactor’s known format set. A bucket that holds v1, v2, and vParquet side by side is a bucket in transit, not a broken bucket.

Under the hood

How to configure it

The block format is selected through the compactor. The ingester inherits it. The block-format-specific compactor key is below:

compactor:
  compaction:
    block_retention: 168h
    compaction_window: 1h
    # vParquet rewritten targets are called vParquet blocks.
    # v2 left in place until rewritten.
    block_search_encoding: v2
    # Bound the per-cycle work so the compactor does not
    # eat the whole bucket in one pass.
    max_compaction_objects: 1000000
    # Flush the trace backend at the cycle boundary.
    flush_blocks_size: 524288000  # 500 MiB

Three keys to call out:

• compactor.compaction.block_search_encoding decides the destination format. Set it explicitly; do not rely on the chart default unless the chart value has been reviewed for the target version.
• compactor.compaction.max_compaction_objects bounds the cycle. A bucket migration that runs unbounded produces a compactor process that pegs a core and blocks the next cycle.
• compactor.compaction.flush_blocks_size matters when the bucket is the destination of a rolling format upgrade. Without it the compactor accumulates rewritten blocks until the cycle ends, doubling the bucket footprint transiently.

The storage block does not need to change for the migration to work. The bucket still holds the same data; only the block format changes.

How to validate it

Severity: READ-ONLY.

1. Inspect the bucket to confirm the format distribution:

aws s3api list-objects-v2 \
  --bucket tempo-traces-prod \
  --prefix tempo/ \
  --output json \
  --query "Contents[].Key" \
  | jq -r '.[]' \
  | head -n 100 \
  | while read k; do
      aws s3 cp "s3://tempo-traces-prod/$k/meta.json" - 2>/dev/null \
        | jq -r '.data_encoding // .version // "unknown"'
    done \
  | sort | uniq -c
# 482 v2
# 18 vParquet

2. Confirm the compactor declares the new format as a known destination:

curl -s http://tempo-compactor:3200/metrics \
  | grep '^tempo_compactor_compaction_block_format'
# tempo_compactor_compaction_block_format{format="v2"} 4
# tempo_compactor_compaction_block_format{format="vParquet"} 1

3. Confirm the compactor is rewriting older blocks:

curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_blocks_compacted_total / {print "compacted:", $2}'
curl -s http://tempo-compactor:3200/metrics \
  | awk '/^tempo_compactor_compaction_errors_total / {print "errors:", $2}'
# compacted: 12842
# errors: 0

4. Verify the read path serves both formats by querying a known trace that is older than the upgrade window:

TRACE=8d3b4e3b3a1c4f5a92a3f3b1d5e0a4f9
curl -sG http://tempo-querier:3200/api/traces/$TRACE | jq '.batches | length'
# 1

5. Confirm a trace stored after the upgrade lands in the new format:

TRACE=$(uuidgen)
# emit, wait for flush, then inspect the block the ingester produced.
aws s3 ls s3://tempo-traces-prod/tempo/ --recursive \
  | grep -E "$TRACE" \
  | awk '{print $4}' \
  | while read key; do
      aws s3 cp "s3://tempo-traces-prod/$key/meta.json" - 2>/dev/null \
        | jq -r '.data_encoding // .version'
    done \
  | sort -u
# vParquet

How it can fail

Five shapes appear in production block-format migrations:

1. Compactor config left at the old default. The chart bumps the binary but the values file still has block_search_encoding: v1. Symptom: the compactor log shows unknown block version vParquet and tempo_compactor_compaction_errors_total rises.
2. Compaction window too long for the bucket size. A bucket with millions of blocks and a compaction_window: 24h will not drain in a week. Symptom: tempo_compactor_blocks_compacted_total rises slowly while the bucket stays the same size.
3. max_compaction_objects too small for the merge set. The compactor cycles but only touches a fraction of the bucket. Symptom: tempo_compactor_blocks_compacted_total resets to zero each cycle while the bucket remains dominated by the old format.
4. Ingest stopped, compactor ignored. A cluster whose ingester was rolled but whose compactor was forgotten still holds a bucket at the old format. Symptom: bucket growth resumes after the cutover but tempo_compactor_blocks_compacted_total is flat for days.
5. flush_blocks_size too small. The compactor flushes early and produces many small merged blocks. Symptom: block count goes up before it goes down. The compactor eventually catches up, but the transient peak is visible on a block-count panel.

How to troubleshoot it

The diagnostic order matters. Each step rules out one failure mode:

1. Is the compactor running? kubectl get pods and the compactor log file. A stopped compactor leaves the bucket frozen at the last cycle.
2. Does the compactor know the new format? The compactor log line emitted at start lists the supported destination formats. If vParquet is missing, the values file is the source of the bug.
3. Is the cycle making progress? Compare tempo_compactor_blocks_compacted_total between two cycles spaced an hour apart. A flat counter means the cycle is blocked.
4. Are errors rising? tempo_compactor_compaction_errors_total. An increase is an actionable signal.
5. Is the bucket format distribution moving? Repeat the aws s3api list-objects-v2 recipe. The old format percentage should drop cycle by cycle.

Security implications

The block format does not change the data. What changes is the metadata emitted and the access pattern:

• Bucket reads scale with block count. A bucket with millions of small blocks produces millions of GetObject calls per query. The IAM policy and the bucket request rate limit must accommodate the fan-out. A migration that produces a transient spike in reads should be throttled by a lower tempo_querier.max_concurrent_queries.
• Metadata size. vParquet rewrites expose the trace id and span id as columns. The block header metadata is unchanged, but the on-disk file carries more tag information. A team with trace-level PII concerns must continue to apply the redaction policy at the ingester, not the compactor.
• Compactor credentials. The compactor needs s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket. A scoped policy that omits s3:DeleteObject on the bucket prefix silently breaks the cleanup pass.

Performance implications

The block-format migration is the cheapest work the compactor does, but it is not free:

• CPU and network. Every merge reads two blocks, writes one, deletes one. The I/O cost scales with bucket size. A compactor CPU budget of one core is sufficient for ten million blocks; a budget of two cores covers the rollout window.
• Memory. vParquet encodes write more compactly per column. A merged block uses roughly half the memory of the source pair. The compactor memory budget does not need to grow.
• Storage transient. During the cycle the compactor holds the rewritten block plus its sources. The bucket footprint roughly doubles until the cleanup pass deletes the sources. This is a transient. Plan a flush_blocks_size large enough to amortise the cleanup.

Production guidance

• Capture the bucket format distribution before the upgrade. The right compactor config is the one that moves that distribution.
• Set compactor.compaction.block_search_encoding explicitly. Do not inherit the chart default silently.
• Bound the per-cycle work with compactor.compaction.max_compaction_objects. The compactor should never cycle the entire bucket in a single pass.
• Wait for tempo_compactor_blocks_compacted_total to match the expected bucket size before declaring the migration done.

Verification

You should now be able to answer:

• What are the three Tempo block format generations, and in which release window was vParquet introduced?
• Which component is responsible for rewriting older blocks into the new format?
• What metric proves that the compactor is making progress on the migration?
• What does the bucket format distribution look like during a healthy migration?
• Why is a stuck compactor invisible on a trace-count panel but visible on a block-count panel?

Quiz