Time Series Storage and the TSDB

The page says the Prometheus volume is at 96 percent. The follow-up questions arrive within the hour: can we keep two years of metrics, can we back Prometheus up, and why did it take four minutes to start after the power cut last night? All three are the same question: how does the TSDB actually store samples? The sysadmin who can answer it sizes the disk once and sleeps; the one who cannot discovers the answers during incidents.

The TSDB is the embedded time-series database inside the Prometheus process. Every sample from every scrape lands there first. There is no external database underneath, and nothing in prometheus.yml configures it — the storage layer is controlled entirely by command-line flags.

What it is

Prometheus storage has three cooperating pieces under --storage.tsdb.path (default data/ in the working directory):

• The head block — the in-memory window that receives all new samples. Roughly the last three hours of data lives here, backed by memory-mapped chunk files.
• The write-ahead log (WAL) — an on-disk append log in wal/. Every sample is logged here before it is considered durable, so a crash costs at most a replay, not the data.
• Persisted blocks — immutable directories, named with ULIDs, each holding a fixed time span of fully indexed data. Blocks are written by the compactor, read by queries, and deleted by retention.

Retention is set by two flags: --storage.tsdb.retention.time (default 15d) deletes blocks older than the limit, and --storage.tsdb.retention.size (default 0, disabled) deletes the oldest blocks when total block size exceeds the budget. Both may be set together; whichever limit a block crosses first wins.

Why a sysadmin cares

Every storage question in production reduces to TSDB mechanics:

• Disk sizing. Disk use is ingestion rate times bytes per sample times retention. Get the inputs and the answer is arithmetic, not guesswork.
• Backup. A safe backup is a filesystem snapshot or a TSDB snapshot — not a naive cp of a live directory.
• Crash recovery. Startup time after a power cut is WAL replay time, and it is proportional to how much was in the head.
• Incident forensics. When queries fail or compactions error, the difference between “delete one block” and “lose everything” is knowing which directory holds what.

How it works

The write path, end to end:

scrape -> parse -> append to head (RAM) + append to WAL (disk)
                        |
            chunk fills (~120 samples) -> new head chunk,
            chunks memory-mapped from chunks_head/
                        |
         head spans ~3h -> compactor cuts the oldest 2h
                           into an immutable block on disk
                        |
         blocks merge: 2h -> 6h -> larger spans
         (largest block = min(31d, retention / 10))
                        |
         block fully outside retention -> deleted

The resulting directory layout:

/var/lib/prometheus/data/
|-- 01J3H8G6E8W0K5Z0Y8Y0J5V4Q0/    block: one time span of data
|   |-- chunks/                    compressed raw samples
|   |-- index                      labels and values to series to chunks
|   |-- meta.json                  block time range, stats, level
|   `-- tombstones                 deletion markers
|-- 01J3HKP2.../                   another block
|-- chunks_head/                   memory-mapped head chunks
|-- wal/
|   |-- 00000127                   WAL segment, about 128 MB each
|   |-- 00000128
|   `-- checkpoint.00000126/       compacted older WAL data
|-- queries.active                 active query tracking
`-- lock                           one server per data directory

Two properties drive most operational behaviour. Blocks are immutable — once written, a block is never modified, only read, compacted into a bigger block, or deleted. And the TSDB does no downsampling: a sample stored at 15s resolution stays at 15s resolution until retention deletes it. Long-term, low-resolution storage is what Thanos and Mimir add on top; Prometheus itself keeps everything at full fidelity.

Out-of-order samples — data arriving with timestamps behind the head — are rejected by default. Since 2.39, Prometheus can accept them within a bounded window when started with --storage.tsdb.out_of_order_time_window (for example 30m); the default remains disabled in 2.55.

How to configure it

Storage flags are command-line only. On a systemd host, put them in an override unit:

# /etc/systemd/system/prometheus.service.d/storage.conf
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/data \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=180GB \
  --storage.tsdb.wal-compression

The empty ExecStart= first is required — systemd overrides replace rather than append. --storage.tsdb.wal-compression (snappy) halves WAL write volume at a small CPU cost and is safe to enable on 2.55.

Disk sizing is the configuration step people skip:

disk_bytes = samples_per_second x bytes_per_sample
             x retention_seconds x headroom

Example: 20,000 samples/s, ~1.3 bytes/sample, 30 days, 20% headroom:
20,000 x 1.3 x 2,592,000 = 67.4 GB  ->  provision at least 80 GB,
                                        set retention.size below it

Measure samples_per_second from the running server rather than estimating it; the validation commands below show how. Set retention.size to roughly 80 percent of the volume so the TSDB itself, not the filesystem, enforces the limit.

How to validate it

# 1. Confirm the flags the running server actually uses.
curl -s http://localhost:9090/api/v1/status/runtimeinfo \
  | jq '.data | {storageRetention, startTime}'

# 2. Confirm the real ingestion rate and active series.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))' \
  | jq '.data.result[0].value[1]'

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_head_series' \
  | jq '.data.result[0].value[1]'

# 3. Confirm on-disk size and block health.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_storage_blocks_bytes' \
  | jq '.data.result[0].value[1]'

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_compactions_failed_total' \
  | jq '.data.result | length'

# 4. Inspect blocks offline (server stopped, or run on a snapshot copy).
promtool tsdb analyze /var/lib/prometheus/data

runtimeinfo is the important first check: because retention is a flag, editing prometheus.yml and reloading does nothing to it, and storageRetention tells you what the process really started with.

How it can fail

1. The volume fills. Co-located logs or an over-long retention eat the disk. Symptom: “no space left on device” in the log, ingestion stops, up stays green, and dashboards flatline into gaps. Prometheus does not emergency-delete outside its retention rules.
2. Unclean shutdown, WAL replay. Power cut or OOM kill. Symptom: startup takes minutes while the log shows WAL replay and repair messages; the last un-replayed segment’s samples may be lost, leaving a short gap. prometheus_tsdb_wal_corruptions_total increments when repair was needed.
3. Block corruption. Failing disk or an incomplete write. Symptom: prometheus_tsdb_compactions_failed_total climbs, and queries touching the block fail with checksum or magic-number errors. The fix is to stop Prometheus, remove the affected block directory, and start again — that block’s time window is gone.
4. Retention misconfigured. retention.time set to 0 disables time-based deletion; without retention.size nothing ever expires. Symptom: prometheus_tsdb_storage_blocks_bytes climbs for months until failure mode 1 arrives.
5. Cardinality explosion. A label change multiplies active series. Symptom: head memory balloons with prometheus_tsdb_head_series, the OOM killer fires, and the server crash-loops through WAL replay on each restart.
6. Out-of-order rejection. A remote-write endpoint or queue flushes old samples after an outage. Symptom: scrape and remote-write errors such as “sample out of order” or “too old”, visible in prometheus_target_scrapes_sample_out_of_order_total, and permanent gaps unless an out-of-order window is configured.

How to troubleshoot it

1. Read the startup log first. WAL replay duration, repair messages, head compaction and truncation lines are all there, in order.
2. Check the effective configuration. runtimeinfo for retention; then df the data volume against prometheus_tsdb_storage_blocks_bytes. TSDB size much larger than expected points at retention; filesystem usage much larger than TSDB size points at something else on the volume.
3. Check the failure counters. compactions_failed_total, wal_corruptions_total, head_truncations_total. A climbing counter names the subsystem.
4. Correlate series with memory. prometheus_tsdb_head_series against resident memory: memory tracking series growth is churn, not a leak.
5. Go offline only when needed. promtool tsdb analyze needs exclusive access — run it against a stopped server or a snapshot copy, never the live directory.
6. Back up before repairing. Take a snapshot (below) before deleting any block or WAL segment.

Security implications

The data directory is the whole database: anyone who can read it can read every label value ever ingested, including instance names and anything sensitive that leaked into labels. Run Prometheus as a dedicated user with the data directory mode-restricted, and treat backups as sensitive artifacts. The admin API — required for server-side snapshots, delete_series and tombstone cleaning — is disabled by default and must be enabled with --web.enable-admin-api; enabling it on an unauthenticated listener hands destructive operations to anyone who can reach the port. Enable it only behind network policy or an authenticating proxy.

Performance implications

Head memory is driven by active series count, not sample volume; the sample rate drives WAL write throughput and compaction work. Compaction rewrites gigabytes in bursts — on slow or shared disks it shows up as IO latency spikes that queries then wait behind. The memory-mapped head means recent-data queries mostly hit page cache, so a host with free RAM serves dashboards faster. Put the data directory on local SSD; network filesystems add latency to WAL appends on the write path and to mmap faults on the read path, and their locking semantics around the lock file are not a safe place to be. Enabling an out-of-order window costs extra head memory for the window it must keep open.

Production guidance

• Set both retention.time and retention.size, with the size limit at about 80 percent of a dedicated volume.
• Size the volume from measured ingestion with the formula above, plus headroom; re-measure after every major cardinality change.
• Back up with the snapshot API (POST /api/v1/admin/tsdb/snapshot with the admin API enabled) or a filesystem snapshot of a quiesced volume; restore by placing the snapshot’s directories into a fresh data path and starting.
• Alert on the storage layer itself: storage_blocks_bytes against volume size, compactions_failed_total, wal_corruptions_total, and WAL replay duration at startup.
• For retention beyond one to two months, plan on Thanos or Mimir rather than multi-year local TSDB; local blocks at full resolution are the wrong tool for years of history.

Verification

You should now be able to answer:

• What are the head block, the WAL and persisted blocks, and in what order does a sample pass through them?
• Which flags control where data lives and how long it is kept, and why can they not be set in prometheus.yml?
• How do you compute the disk a given ingestion rate and retention need?
• What does Prometheus do after an unclean shutdown, and what does WAL replay cost?
• How do you back up a running server, and why is cp of the live data directory not a backup?

Quiz