The TSDB Engine

A kernel panic takes down the Prometheus host mid-scrape. Sixty seconds after the restart, the server is answering queries over the last three hours of metrics as if nothing happened, and not a byte of extra disk was consumed by the crash. That behaviour — crash safety with no external database — is the TSDB: the embedded time-series engine inside every Prometheus server. It is also where your disk, your memory and your retention policy actually live.

What it is

The TSDB is a local, embedded, single-writer store. Every sample from every scrape lands here first (lesson 02), and everything Prometheus knows how to answer comes back out of it. There is no pluggable storage backend; if you outgrow one server’s disk, the answer is remote write (lesson 06), not a different TSDB.

The layout on disk

/var/lib/prometheus/            # --storage.tsdb.path (default: ./data)
├── lockfile                    # one server per data dir; do not share it
├── wal/                        # write-ahead log, 128 MiB segments
│   ├── 00000042
│   └── checkpoint.00000041/    # WAL rewritten after head truncation
├── chunks_head/                # mmapped chunks backing the head block
├── 01J8X3HN…/                  # a persisted block, named by ULID
│   ├── chunks/000001           # compressed samples, segments up to 512 MiB
│   ├── index                   # symbols, series, postings: label → series
│   ├── meta.json               # minTime, maxTime, stats, compaction level
│   └── tombstones              # soft-deleted series (the delete API)
└── snapshots/                  # admin-API snapshots (hard links)

How it works

The head block. Every appended sample goes to the head, an in-memory structure covering roughly the last three hours (1.5× the two-hour block range), and to the write-ahead log on disk. On crash, Prometheus replays the WAL to rebuild the head. The head’s chunks are memory-mapped from chunks_head/, so the heap holds series metadata while sample data rides the OS page cache — which is why “Prometheus memory usage” is really “page cache plus head series”.

Blocks. About every two hours of data, the head is compacted into a persisted block: an immutable directory with compressed sample chunks (typically 1–3 bytes per sample), an inverted index, meta.json describing the block’s time range and statistics, and tombstones for deleted series. Immutable blocks are what make snapshots and retention cheap: nothing inside them ever changes.

Checkpoints and truncation. When the head block is cut, the head is truncated and the WAL is checkpointed: the WAL is rewritten without the records that are now persisted in the new block (the checkpoint.* directory), and the old segments are deleted. A checkpoint is why a crash replays minutes of WAL, not days.

Compaction. A background compactor runs every two hours and does two different jobs. Horizontal compaction merges adjacent blocks into larger time windows — 2h, then 6h, then 18h, growing exponentially, capped at 10% of retention time or 31 days, whichever is smaller. Vertical compaction merges overlapping blocks into one — the situation created by HA replica data or backfilled imports. Neither job ever touches sample values. The TSDB does not downsample. Retention is the deletion of whole blocks, not the aggregation of old data; a 30-day-old graph has the same resolution as a 30-minute-old one.

Retention. Two knobs: --storage.tsdb.retention.time (default 15d) and --storage.tsdb.retention.size (default disabled). Both are enforced at block granularity — a block is deleted when it falls entirely outside retention. Size retention is a ceiling, not a guarantee: the engine deletes the oldest blocks to get back under the limit.

Queries and mmap. Queries read the head plus the relevant blocks, all memory-mapped. Big dashboard queries show up as page cache pressure, not heap growth — which is why Prometheus coexists badly with other memory-hungry processes on the same host.

What changed in 2.55. Honestly: nothing about this layout. The on-disk block format has been stable since the 2.0 redesign, and a 2.55 server reads blocks written by much older 2.x releases. The 2.55-era work went into ingestion paths (the OTLP receiver), native histograms and PromQL performance. Operationally, this lesson is not version-fragile.

How to configure it

Storage is configured with flags, not YAML:

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=800GB \
  --storage.tsdb.wal-compression \
  --web.enable-admin-api        # required for the snapshot and delete APIs

Set both retention knobs on any disk-constrained host: time for policy, size as the circuit-breaker. The admin API is needed for snapshots — enable it deliberately, because it also enables deletion (see security below).

How to validate it

# The engine's own status: head stats and cardinality leaders
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.headStats | {numSeries, chunkCount, minTime, maxTime}'

# Blocks on disk: count and time spread
ls /var/lib/prometheus | grep -c '^01'

# Take a snapshot (admin API required)
curl -s -X POST localhost:9090/api/v1/admin/tsdb/snapshot
# {"status":"success","data":{"name":"20260813T091500Z-1c4f..."}}
ls /var/lib/prometheus/snapshots/

Engine metrics worth graphing or alerting on:

prometheus_tsdb_head_series                  # live series in the head block
prometheus_tsdb_compactions_total            # should climb steadily
prometheus_tsdb_compactions_failed_total     # should be flat at zero
prometheus_tsdb_wal_corruptions_total        # should be flat at zero
prometheus_tsdb_head_truncations_total       # head cuts, roughly 2-hourly
prometheus_tsdb_compaction_duration_seconds  # compaction cost

How it can fail

1. Disk full, no size retention. Symptom: prometheus_tsdb_compactions_failed_total climbs, “no space left on device” in the logs, ingestion stalls, and the server can crash-loop as the WAL grows unreclaimable.
2. WAL corruption after power loss. Symptom: a slow start, “WAL corruption” log lines, and a small window of lost recent data. Prometheus truncates at the corrupt record and carries on; prometheus_tsdb_wal_corruptions_total records it.
3. Overlapping blocks — a botched restore, a manual block copy, or two servers sharing a directory. Symptom: “overlapping blocks” compaction errors, failed compactions, an ever-growing block count and slowing queries.
4. Size retention set too small. Symptom: historical queries silently return only a few days; the engine dutifully deleted the blocks you wanted. It did exactly what you asked.
5. Snapshot job without the admin API. Symptom: HTTP 403 “admin API disabled” — and a backup cron that has been doing nothing for months. Check for the snapshot directory, not the cron exit code.
6. OOM-kill restart loop. Symptom: every restart replays the full WAL (“replaying WAL” in the logs for minutes), scrapes gap, the server gets killed again mid-recovery. The trigger is memory pressure, but the visible cost is WAL replay time.

How to troubleshoot it

1. Is it running and ready? curl localhost:9090/-/ready; the logs show how long WAL replay took.
2. Disk. df on the TSDB path, du -sh wal/ and the block directories. Full or nearly full explains most engine incidents.
3. Engine metrics. Failed compactions? WAL corruptions? Head series exploding? Each points at a different failure above.
4. Cardinality leaders. /api/v1/status/tsdb returns seriesCountByMetricName — the top offenders when the head series count balloons.
5. Logs. Grep for “compact”, “overlapping”, “corruption”, “checkpoint”. The engine narrates its own failures clearly.
6. Before any surgery: snapshot (or stop the server and copy the data directory). Never delete WAL segments by hand unless you accept losing everything in them.

Security implications

• The data directory contains every metric and every label — hostnames, usernames, environment names, occasionally worse. 0700/0750 owned by the prometheus user; nothing else needs access.
• The admin API (--web.enable-admin-api) can snapshot, delete series, and write tombstones, unauthenticated, over HTTP. Bind it to localhost or put an authenticating proxy in front; the security part of this course covers the pattern.
• Snapshots inherit the sensitivity of the data. Encrypt the offsite copy and restrict who can restore it.

Performance implications

Disk usage is roughly 1–3 bytes per sample on average, dominated by your series count and scrape interval — size retention with retention.size on any disk smaller than your growth curve. Memory tracks active head series, not total stored data; churn (short-lived series) is the expensive shape. Compaction is an I/O burst every two hours; on starved disks it coincides with query latency spikes, which is normal, not a fault. WAL compression trades a little CPU for meaningful disk-write savings.

Verification

You should now be able to answer:

• What lives in the WAL, in the head block, and in a persisted block — and in what order does a sample visit them?
• What is a WAL checkpoint, and when is it written?
• Does compaction ever reduce the resolution of old data?
• What are the two retention knobs, and at what granularity do they act?
• How do you take a consistent backup without stopping scrapes, and how do you restore it?

Quiz