Retention and Storage

Every Prometheus eventually meets its disk. The only choice is whether the meeting is planned — a retention policy you computed and alerts that fire at 80% — or unplanned: a 03:00 page because the monitoring host ran out of space and took its own alerting down with it. Retention looks like a boring capacity topic. It is actually the difference between a monitoring system that degrades gracefully and one that fails at the worst moment.

What retention actually deletes

Lesson 02 established the shape of the data directory: an in-memory head holding the newest two to three hours, a WAL journalling it, and immutable block directories holding compacted history. Retention applies to exactly one of those: the blocks. Prometheus deletes a block directory when the block’s entire time range falls outside the retention limit. Two consequences worth memorising:

• Deletion is block-granular. The oldest sample on disk can be older than your configured retention, by up to the span of the largest block, because a block that straddles the boundary is kept whole until it lies fully outside it.
• Retention never touches the head or the WAL. Recent data is governed by head truncation and WAL checkpointing, not by your retention flags.

The two limits

Prometheus 2.55.x offers two independent limits:

• --storage.tsdb.retention.time — how long to keep samples. Default 15d. Accepts durations like 30d, 4w, 1y. Setting it to 0 disables the time limit (leaving size in charge) — legal, rarely wise.
• --storage.tsdb.retention.size — the maximum number of bytes to keep in storage blocks, e.g. 450GB. Units are required and are powers of two (1KB is 1024 bytes). Default 0, meaning disabled. Still marked experimental in 2.55.x — and still the seatbelt you want, for reasons below.

Set both, and whichever limit is reached first wins: blocks age out at 30 days or when the block budget is exceeded, whichever happens sooner. That interaction is the whole design pattern — time expresses intent, size expresses physics.

Sizing the disk

The rule of thumb that survives contact with production: a sample costs 1-2 bytes on disk after compression. Plan at 2; be pleasantly surprised at 1.3.

bytes needed  ≈  samples/sec  ×  seconds of retention  ×  bytes per sample

worked example, 100,000 samples/sec, 30 days:
  100,000 × 2,592,000 s = 2.59e11 samples
  at 1.5 bytes/sample   →  ~390 GB
  at 2.0 bytes/sample   →  ~520 GB   (planning figure)

add 25-30% headroom for compaction scratch, WAL and head:
  provision a ~700 GB volume

Two honest caveats. First, the 1-2 byte figure assumes mostly well-behaved counters and gauges; high-churn series and sparse histograms cost more. Second, ingestion is not static: every new exporter, every new service, every label mistake moves the rate. Re-run the maths whenever scrape configuration changes materially, and treat the number you measure today as the floor.

Configuring retention

Flags live in the unit (lesson 03), so retention changes are flag changes: they need daemon-reload and a restart, not a reload.

# /etc/systemd/system/prometheus.service.d/20-retention.conf
# The empty ExecStart= clears the base unit's command before restating it.
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=450GB \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.enable-lifecycle

# CONFIGURATION + SERVICE-IMPACT -- apply flag changes
sudo systemctl daemon-reload
sudo systemctl restart prometheus

On a 500 GB volume, 450GB of block budget plus head, WAL and compaction scratch lands you at roughly the utilisation you planned — not at a full disk with a smug configuration file.

Measuring reality

Do not trust the spreadsheet; the server tells you the truth:

# READ-ONLY -- retention flags the running process actually uses
curl -s http://localhost:9090/api/v1/status/flags \
  | jq -r '.data["storage.tsdb.retention.time"], .data["storage.tsdb.retention.size"]'
# 30d
# 450GB

# READ-ONLY -- measured ingestion, samples per second (1h average)
curl -sg http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(prometheus_tsdb_head_samples_appended_total[1h])' \
  | jq -r '.data.result[0].value[1]'
# 102341.71

# READ-ONLY -- active series in the head
curl -sg http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series' \
  | jq -r '.data.result[0].value[1]'

# READ-ONLY -- oldest data currently held (unix seconds; compare with intent)
curl -sg http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_lowest_timestamp' \
  | jq -r '.data.result[0].value[1]'

# READ-ONLY -- block inventory (run against a snapshot, or a stopped
# server; a live data directory is locked)
sudo promtool tsdb list /var/lib/prometheus | head -5

# READ-ONLY -- the boring, decisive ones
df -h /var/lib/prometheus
du -sh /var/lib/prometheus/*

prometheus_tsdb_lowest_timestamp against your configured retention is the single most informative check: if the oldest data is much newer than 30 days ago, the size limit or the disk is silently overriding your intent.

When the disk fills

Prometheus has no emergency self-truncation switch. When the filesystem is full, WAL appends fail, head ingestion stalls, scrapes start timing out, and the log fills with no space left on device. Compaction — the mechanism that would eventually free space — itself needs scratch space, so a full disk deadlocks the recovery path too. The exact failure surface varies by version, but the safe operational statement is: a full TSDB disk is a monitoring outage, and the platform cannot alert on it because the platform is what is down. Free space (or grow the filesystem online), start the service, and expect a WAL replay before /-/ready answers. This is why the alert chain on disk usage — 70%, 80%, 90% — belongs on a different system than the one being watched, and why retention.size exists at all.

Long-term data: remote storage

Local TSDB is hot storage: fast to query, sized for weeks. History measured in months or years belongs in an object-storage-backed system — Grafana Mimir, Thanos, Cortex, or a managed service — fed by remote_write:

# /etc/prometheus/prometheus.yml -- the long-term leg
remote_write:
  - url: https://mimir.example.org/api/v1/push
    queue_config:
      max_samples_per_send: 1000

The pattern: keep 15-30 days local for alerting and dashboard latency, stream everything to the remote system for the long tail. Two honest caveats: remote_write is a stream, not a backup — it does not help you restore local history; and during a remote outage the WAL buffers the backlog for hours, not days — a long outage silently drops remote history. Alert on remote-write lag (prometheus_remote_storage_highest_timestamp_in_seconds falling behind the WAL) as its own failure mode.

How it can fail

1. Time-only retention plus a cardinality explosion. A bad label doubles ingestion; 30 days of data doubles in size; the disk fills weeks before the time limit would have protected you. This is the argument for always setting the size flag.
2. Trusting retention.size as a hard cap. WAL and head are outside the budget; a churn storm grows them until the disk fills anyway. The flag is a budget for blocks, not a circuit breaker for the filesystem.
3. Retention longer than the maths supports. The disk fills before 30 days elapse; oldest data is silently newer than intent. prometheus_tsdb_lowest_timestamp exposes it.
4. Full disk deadlock. WAL writes fail, ingestion stalls, and compaction cannot free space because compaction needs space. Recovery is manual: free space, restart, wait for WAL replay.
5. A corrupted block halts compaction. The log repeats compaction errors and prometheus_tsdb_compactions_failed_total climbs; disk usage grows unbounded because new blocks never merge and old ones never delete. Inspect with promtool tsdb, remove the offending block directory, restart.
6. Boundary confusion. An engineer queries day 31 of a 30d retention and finds data — a 31-day-capped block has not fallen fully outside the window yet. Not a bug; block granularity. The reverse panic — “data vanished at day 28” — is mode 3, and worth distinguishing.

How to troubleshoot it

Disk-pressure investigations, in order:

1. How full, and where does it go? df -h on the TSDB mount, then du -sh /var/lib/prometheus/*. WAL bigger than a few hours of ingest, or one huge block, each tells a different story.
2. What is the measured ingest? rate( prometheus_tsdb_head_samples_appended_total[1h]) and prometheus_tsdb_head_series. A step change in either names the day the growth started; correlate with deploys.
3. Is retention even applied? /api/v1/status/flags for the running values, and prometheus_tsdb_lowest_timestamp for the effective horizon.
4. Is compaction healthy? prometheus_tsdb_compactions_failed_total rising, or repeated compaction failed lines in journalctl -u prometheus, point at a corrupt block.
5. Grow, do not restart-shrink. On LVM, lvextend plus xfs_growfs (or resize2fs) grows the filesystem online and Prometheus sees the space immediately. Deleting blocks by hand while the server runs is the move that turns a capacity problem into a corruption problem — do not.

Security implications

Retention is a compliance decision, not just a capacity one: it defines how long behavioural records — which may carry user-identifying labels — persist, and legal or policy frameworks may set both a minimum and a maximum. Deletion via the admin API requires --web.enable-admin-api, a deliberate choice with its own abuse potential. Snapshots and backups contain the full TSDB; protect them with the same care as the live data. And the remote_write block holds credentials for the long-term store — one more reason prometheus.yml stays at 0640 (lesson 06).

Performance implications

Longer retention means more blocks, larger indexes, slower long-range queries and more page-cache pressure; the honest cost of a year of local data is RAM and query latency, not just disk. Compaction is an I/O burst you can watch in iostat; on spinning disks it is the argument for keeping the TSDB off the OS spindle. Remote write costs a little CPU and a steady network stream proportional to ingest. None of these is a reason to skimp on retention — they are the inputs to the sizing maths, and to the decision of what goes remote.

Production guidance

• Set both flags, always. Time expresses intent; size protects the host. Derive size at roughly 70% of the TSDB volume.
• Provision with 20-30% free space for compaction scratch, WAL and head — and alert at 70/80/90% from a system that does not depend on this Prometheus.
• Re-run the sizing maths after every material scrape change; record the current samples/sec figure in the runbook.
• Keep local retention short (15-30d) and send the long tail to remote storage; do not buy years of local SSD to answer once-a-quarter questions.
• Review prometheus_tsdb_lowest_timestamp against intent in the quarterly capacity review.

Verification

You should now be able to answer:

• Which parts of the data directory does retention.size budget, and which does it ignore?
• Your server ingests 100k samples/sec and you want 30 days. Walk the sizing maths.
• What does Prometheus do when the TSDB filesystem fills, and why can compaction not rescue it?
• Why can the oldest sample on disk be older than retention.time?
• What is the standard pattern for keeping years of queryable history, and what does the WAL buffer during a remote outage?

Quiz