TSDB Migration

A Prometheus 2.55 upgrade is in flight. The new binary accepts the old TSDB on first start; over the next several minutes the write-ahead log plays back into the new layout. The operator is careful: a snapshot was taken an hour before, the previous binary’s image is still tagged in the deployment manifest, and the WAL replay completes without errors. By 03:10 the new binary is serving traffic; by 03:18 the first block compactor run writes a block in the new format.

This is the happy path. The lesson exists because it does not always take this shape. When the on-disk format changes between versions, three failure modes follow the same blueprint: a snapshot is missing, the WAL replay completes but a later compaction fails, or the operator rolls the binary back and discovers the rollback binary cannot read what the new binary wrote.

What it is

A TSDB migration is the operation that moves Prometheus from one on-disk format to another. The format covers:

  +-------------------------+--------------------------------------------+
  | layer                   | description                                |
  +-------------------------+--------------------------------------------+
  | write-ahead log (WAL)   | segment format, record layout,             |
  |                         | compression, truncation semantics          |
  | compacted blocks        | index format, chunk format, meta.json      |
  |                         | schema, mmap layout                        |
  | in-memory head block    | head chunk format, series hash semantics   |
  +-------------------------+--------------------------------------------+

The WAL block lifecycle is the most common migration point between recent minors. In Prometheus 2.55 the WAL block lifecycle is the supported operation; earlier 2.x releases used a simpler WAL format. Operators moving from pre-2.55 binaries into 2.55 need to understand the boundary: the old WAL plays forward cleanly into the new layout, but the new layout is not readable by the previous binary.

The on-disk format version is recorded inside each block’s meta.json and inside each WAL segment’s header. Operators can inspect it without a running process.

Why a sysadmin cares

A failed TSDB migration is the most expensive Prometheus incident you can have. Three failure shapes recur:

1. No snapshot available. The operator skipped the snapshot step because the upgrade is “just a minor.” The new binary writes a block the old binary cannot read. Rollback becomes “lose the in-window samples” or “replay from remote.”
2. WAL replay crashes mid-way. A disk fault or a memory pressure event interrupts the WAL replay. The new binary refuses to start with the partial state. Recovery is from snapshot, not from in-place repair.
3. Downgrade after upgrade. The new binary wrote a block in the new format. The operator rolls the deployment back to the previous image. The old binary refuses to read the new block. Rollback is blocked until the previous-binary data/ directory is restored.

The discipline in this lesson prevents each of those three.

How it works

The migration shape has three phases. Each phase has a defined pass / fail criterion.

  phase 1: snapshot          PASS = restorable tarball exists
       |
       v
  phase 2: WAL replay        PASS = replay completes, head reports
       |                       new minTime, no parse errors in log
       v
  phase 3: first compaction  PASS = first block in new format written
                                to blocks/, old WAL truncated

Phase 1 produces the safety net. Phase 2 is the actual migration; it happens automatically on first start of the new binary against the old data directory. Phase 3 is the irreversibility point — the block in the new format makes downgrade-unfriendly.

How to configure it

The migration is governed by --storage.tsdb.wal-compression (added in 2.55.x), --storage.tsdb.retention.*, and the operational decisions around snapshot timing and storage.

Migration-flag configuration

# /etc/prometheus/prometheus.yml — flag set on the 2.55 binary
spec:
  containers:
    - name: prometheus
      image: prom/prometheus:v2.55.1
      args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=30d
        - --storage.tsdb.wal-compression
        - --storage.tsdb.min-block-duration=2h
        - --storage.tsdb.max-block-duration=25h
      volumeMounts:
        - name: storage
          mountPath: /prometheus

--storage.tsdb.wal-compression enables the new WAL block lifecycle that the previous binary did not emit. The flag is on by default in 2.55.x; the explicit setting documents the choice in the deployment manifest.

Pre-upgrade snapshot script

#!/usr/bin/env bash
# Pre-upgrade snapshot script. Run from the operator workstation;
# expects the API URL via $PROM_API.

set -euo pipefail

: "${PROM_API:?PROM_API must be set, e.g. http://prom-0:9090}"
: "${SNAP_DIR:?SNAP_DIR must be set, e.g. /srv/backups/prom/2026-08-14}"

TS=$(date -u +%Y%m%dT%H%M%SZ)
TARGET="${SNAP_DIR}/${TS}"

mkdir -p "${TARGET}"

# DATA-LOSS-RISK: snapshot the TSDB through the API.
# This is the supported path; do not copy the data directory
# directly under load.
curl -fsS --data-urlencode "skip_head=true" \
  "${PROM_API}/api/v1/tsdb/snapshot" \
  | jq -r '.data.name' \
  | xargs -I {} rsync -a "${PROM_API_HOST}:/prometheus/snapshots/{}/" "${TARGET}/"

# Verify the snapshot is readable.
ls -lh "${TARGET}" | head
echo "snapshot OK at ${TARGET}"

The script asks the running Prometheus to take a snapshot, copies the resulting directory off-host, and prints a one-line verification. skip_head=true keeps the WAL replay window small on the operator’s restore path.

Restart order for a single-replica deployment

# CONFIGURATION: drain scrape traffic on the canary first.
kubectl label pod prom-0 app=prometheus-canary --overwrite

# SERVICE-IMPACT: stop, snapshot, restart.
kubectl exec prom-0 -- sh -c 'kill -TERM $(pidof prometheus) && \
  while pidof prometheus >/dev/null; do sleep 1; done'
kubectl exec prom-0 -- curl -fsS --data-urlencode 'skip_head=true' \
  http://localhost:9090/api/v1/tsdb/snapshot \
  | jq -r '.data.name'
kubectl exec prom-0 -- rsync -a /prometheus/snapshots/ /srv/snapshots/2026-08-14/

# Validate the snapshot is intact (see lesson 06).
kubectl exec prom-0 -- tar -czf /tmp/snap.tar.gz -C /srv/snapshots/2026-08-14 .

How to validate it

Five mechanical checks confirm the migration succeeded and the pre-conditions for safe rollback still hold.

# READ-ONLY: confirm version.
curl -fsS http://prometheus.internal:9090/api/v1/status/runtimeinfo \
  | jq '.data.version'
# "2.55.1"

# READ-ONLY: confirm WAL replay completed; head reports live range.
curl -fsS http://prometheus.internal:9090/api/v1/status/tsdb \
  | jq '.data.headStats'
# {"numSeries":42118,"minTime":"...","maxTime":"..."}
# minTime and maxTime must differ; equal values mean replay did
# not finish.

# READ-ONLY: confirm new format blocks have been written.
ls -1 /prometheus/blocks | head
# 01HMR4X3G5R8S9V3N5P6Q7R8S
# 01HMR4X4J...
# Each directory's meta.json must parse.
for d in /prometheus/blocks/*/; do
  jq -e '.version' "${d}/meta.json" >/dev/null || echo "FAIL: ${d}"
done
# (no output means all blocks parsed)

# READ-ONLY: confirm the new WAL flag is in effect.
curl -fsS http://prometheus.internal:9090/api/v1/status/config \
  | grep -- '--storage.tsdb.wal-compression' || echo "flag absent"

# READ-ONLY: confirm snapshot artifact is restorable.
tar -tzf /srv/snapshots/2026-08-14/snap.tar.gz | wc -l
# (matches the expected block + wal count from before the upgrade)

A clean migration: the version is 2.55.x; the head shows a live range; new blocks have been written and parse; the WAL flag is on; the snapshot is restorable from tar.

How it can fail

Six shapes recur. The symptom is the lesson; the fix follows.

1. Snapshot directory on the same disk. The snapshot is written to /prometheus/snapshots/ on the same mount as the data. A disk fault takes both. The fix is to copy the snapshot off-host before the upgrade.
2. WAL replay OOM. The replay rate exceeds the memory the container was sized for. The new binary crashes on startup with Out of memory. The fix is to pre-size the container to the replay peak and to monitor prometheus_tsdb_head_series before the upgrade.
3. Compaction crash on first run. The first compaction after the upgrade writes a block in the new format and fails midway. The partial block confuses subsequent runs. The fix is to remove the partial block dir and let the compactor rebuild it.
4. Old binary started against new TSDB. The deployment rolled back to the pre-2.55 image. The old binary refuses to start with cannot read block format version 2. The fix is to restore the data directory from the snapshot taken before the upgrade.
5. Remote-write backpressure and WAL growth. During a long WAL replay the remote backend may reject samples older than its retention window. The WAL fills. The fix is to pause remote-write during the replay window or to size the WAL to the replay duration.
6. Skip-head snapshot becomes the rollback target. A skip_head=true snapshot does not include in-flight WAL samples. If the operator’s “rollback” is “restore snapshot, restart old binary,” the in-flight samples are lost. The fix is to take a non-skip snapshot if the goal is full recovery.

How to troubleshoot it

The diagnostic order when a migration has failed:

1. What does the log say? kubectl logs ... | grep -E 'parse error|migration|out of memory|wal replay'. The first level=error line after the start time is the closest indicator of what failed.
2. What does the head say? curl /api/v1/status/tsdb. A minTime = maxTime indicates the replay did not finish. A missing /api/v1/status/tsdb response indicates the process refused to start.
3. What does the WAL look like? ls -la /prometheus/wal/. A directory that contains only one or two segments for a Prometheus that has been running for hours indicates truncation is disabled (often by a long retention run).
4. What does the most recent block look like? ls -la /prometheus/blocks/$(ls -t /prometheus/blocks | head -1). If the directory exists but meta.json does not parse, the compactor crashed mid-block. Remove the dir and continue.
5. Is the snapshot intact? tar -tzf snap.tar.gz | wc -l and tar -xzOf snap.tar.gz snap/meta.json | jq. If the snapshot itself is corrupt, recovery becomes remote-write replay only.

Security implications

The migration has no direct security implications beyond the general posture of the data directory:

• The snapshot directory inherits the data-directory permissions. A chmod 0700 /prometheus is the operator’s responsibility, not the binary’s.
• Remote-write credentials are unchanged by the upgrade but may be rotated by the remote backend’s policy. Verify remote-write is still flowing after the replay.
• The HTTP API exposes block-level metadata. New diagnostic endpoints added in recent releases may surface internal identifiers. Restrict the API ACL if the previous posture relied on the older endpoint set.

Performance implications

The performance cost of the migration is concentrated in two windows:

• WAL replay. Reads the WAL in order at disk speed. A 30-day retention Prometheus with several thousand series replays in minutes. A 1-year retention Prometheus with millions of series may replay for over an hour.
• First compaction. Writes the first block in the new format. I/O-bound on the data directory. Visible as a one-time spike in disk write IOPS.

The readiness probe holds the new replica out of rotation during both windows. Do not size the probe period shorter than the replay peak (a 5-second probe on a 90-second replay will recycle the container).

Verification

You should now be able to answer:

• Which on-disk layers does a TSDB migration change between Prometheus versions?
• What is the operator’s recovery path if WAL replay crashes mid-migration?
• Why is a skip_head=true snapshot an unsafe rollback target for a full-recovery posture?
• How do you verify that the first compaction has actually written in the new format?

Quiz