Break/Fix: Prometheus Disk Full

Reported symptoms

Prometheus stopped ingesting at 03:47. /-/ready returns 503, every dashboard flatlines at 03:44, and the platform is now blind to the rest of the estate as well as to itself.

Nothing paged. The disk-space alert exists, has existed for two years, and is green as you read this.

Everything a runbook would tell you to check is clean:

• Retention has not been touched in eight months. The unit still carries --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=380GB, and the file’s modification time agrees.
• There has been no cardinality event. Head series has sat at about 1.9M for four months and ingestion at about 62,000 samples per second.
• The capacity dashboard reads 30 percent used. It has read 30 percent every day for a year and a half, including this morning.
• The nightly backup job has reported success for sixty-four consecutive nights, and the alert on the age of the newest object in S3 is green.

The on-call restarted Prometheus, which is the obvious first move and was the wrong one. It replayed the WAL for twelve minutes, during which the historical data could not be queried either, and came back to the same full volume.

Evidence provided

$ df -h /var/lib/prometheus /

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    500G  500G     0 100% /var/lib/prometheus
/dev/nvme0n1p2   80G  4.6G   72G   6% /

$ du -sh /var/lib/prometheus/data/snapshots /var/lib/prometheus/data/wal /var/lib/prometheus/data/chunks_head

489G    /var/lib/prometheus/data/snapshots
4.1G    /var/lib/prometheus/data/wal
7.2G    /var/lib/prometheus/data/chunks_head

$ ls /var/lib/prometheus/data/snapshots | wc -l && du -sh /var/lib/prometheus/data/snapshots/* | head -4

64
150G    /var/lib/prometheus/data/snapshots/20260616T020014Z-3f9a1c
5.4G    /var/lib/prometheus/data/snapshots/20260617T020012Z-77b204
5.3G    /var/lib/prometheus/data/snapshots/20260618T020019Z-1ad9e0
5.5G    /var/lib/prometheus/data/snapshots/20260619T020015Z-c04e33

$ stat -c '%h %n' /var/lib/prometheus/data/snapshots/20260616T020014Z-3f9a1c/01J*/chunks/000001 | head -3

51 /var/lib/prometheus/data/snapshots/20260616T020014Z-3f9a1c/01J8QK4T2R/chunks/000001
64 /var/lib/prometheus/data/snapshots/20260616T020014Z-3f9a1c/01J8VB9M7C/chunks/000001
64 /var/lib/prometheus/data/snapshots/20260616T020014Z-3f9a1c/01J8ZC1H5X/chunks/000001

Prometheus’s own view of its storage, taken from the long-term store because the local instance is down:

prometheus_tsdb_storage_blocks_bytes    # ~1.61e+11, flat for months
prometheus_tsdb_time_retentions_total   # climbing daily, all sixty-four days
prometheus_tsdb_size_retentions_total   # 0

The disk alert, unchanged since it was written:

- alert: NodeDiskSpaceLow
  expr: |
    node_filesystem_avail_bytes{mountpoint="/"}
      / node_filesystem_size_bytes{mountpoint="/"} < 0.10
  for: 15m
  labels:
    severity: critical

And the backup job, in the crontab at 02:00 nightly:

#!/usr/bin/env bash
set -euo pipefail

PROM_URL="http://localhost:9090"
SNAP_ROOT="/var/lib/prometheus/data/snapshots"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"

SNAP_NAME=$(curl -fsS -X POST "${PROM_URL}/api/v1/admin/tsdb/snapshot" \
  | sed -E 's/.*"name":"([^"]+)".*/\1/')

tar -C "${SNAP_ROOT}" -czf "/tmp/${SNAP_NAME}.tar.gz" "${SNAP_NAME}"

aws s3 cp "/tmp/${SNAP_NAME}.tar.gz" \
  "s3://prom-backup-primary/${STAMP}/prometheus.tar.gz"

# Clean up the local snapshot now that the tarball is in S3.
rm -f "/tmp/${SNAP_NAME}.tar.gz"

Work the evidence before reading on

Start from the one fact that should be impossible. Retention has been running correctly, every day, for the entire period, and the volume filled anyway.

1. prometheus_tsdb_time_retentions_total has climbed daily for sixty-four days. What does that counter actually count, and what does a Prometheus block deletion do at the filesystem layer?
2. du -sh on the snapshots directory says 489G, on a 500G volume. Listing the directories individually credits 150G to the oldest and about 5G to each of the other sixty-three, and those numbers sum back to 489G rather than exceeding it. What does du do with an inode it has already seen in the same invocation, and what does that arrangement of numbers tell you about what these directories contain?
3. A chunk file has a link count of 51. Where are the other fifty links, and what happens when one of them is removed?
4. prometheus_tsdb_storage_blocks_bytes has been flat and correct all along. Which bytes does that gauge count, and which bytes on this volume does it have no way of knowing about?
5. The comment in the backup script says it is cleaning up the local snapshot. Read the command underneath it. Which file does it actually remove?
6. --storage.tsdb.retention.size=380GB was set as a safety net and never triggered. What total does that flag compute, and is the 489G part of it?

Before continuing: name the single line the backup script is missing, and say why the disk grew in a straight line rather than in steps.

Root cause

A snapshot is a set of hard links, and a hard link is a retention policy

The admin snapshot API does not copy the TSDB. It hard-links each persistent block file into a new directory under data/snapshots/ and writes out one genuinely new block containing the head data. That is what makes the operation fast and, on the night it runs, nearly free.

Read the per-directory listing carefully, because du counts each inode once per invocation and therefore credits every shared file to whichever directory it reached first. The oldest snapshot is charged 150G - the thirty days of blocks that were live when it was taken - and each later one is charged only about 5G, which is the single day of blocks that nothing listed before it already linked. Those numbers are not the cost of each snapshot. They are the marginal cost of each snapshot given all the ones before it, and they sum to 489G because between them the sixty-four directories hold a link to every block written since the job was enabled.

Retention deletes a block by unlinking its files. Unlinking a file that still has other links does not free anything; it decrements a counter. So from the first night this job ran, every block that retention aged out kept its bytes on the volume, held there by a link inside a snapshot directory that nothing was ever going to remove.

The script explains itself. The comment says it is cleaning up the local snapshot; the command removes /tmp/${SNAP_NAME}.tar.gz, which is the staging tarball. ${SNAP_ROOT}/${SNAP_NAME} - the directory holding the links - is never touched. One line, absent for sixty-four nights.

The disk grew in a straight line, which is why the growth looked like use

There is no knee anywhere on the graph, and people looked for one. The reason is that the pinning started immediately. The first snapshot linked all thirty days of live blocks; retention deleted the oldest of them the same day and freed nothing; and from that day forward the occupied window’s left edge was frozen while its right edge advanced at the block-creation rate. The volume therefore grew at almost exactly the rate the platform ingests, from day one, in a perfectly straight line.

A straight line at the ingestion rate is indistinguishable from healthy growth if you are not also asking whether anything is being deleted. Worse, a plausible explanation was available: a new cluster was onboarded three weeks into the sixty-four days, and the growth already underway was attributed to it.

Both brakes measured the TSDB’s opinion, and the TSDB was right

prometheus_tsdb_storage_blocks_bytes counts the persistent blocks the TSDB manages. Once retention deletes a block, it stops being a block the TSDB manages, so its bytes leave the gauge on the day they stop being freeable. The gauge was accurate at 150 GiB the entire time. The capacity dashboard built on it was accurate. Both were answering a question nobody meant to ask.

--storage.tsdb.retention.size was blind for the same reason. It counts the WAL and the m-mapped head chunks toward its total and deletes only persistent blocks, so its arithmetic was 150G of blocks plus 11G of WAL and head against a 380G ceiling. It had no cause to act and never did: prometheus_tsdb_size_retentions_total is still 0.

The filesystem alert would have caught all of it, and it has been pointed at the wrong volume since /var/lib/prometheus was moved onto its own device during a resize eighteen months ago. It has been passing ever since, on a filesystem that is six percent full.

Resolution

1. Confirm the snapshots are redundant before deleting any of them. The S3 objects are the copy of record; if you cannot confirm the uploads, keep the newest snapshot and treat the confirmation as incident work rather than as follow-up.
2. Remove the oldest snapshot directories one at a time, checking df after each. Deleting all sixty-four in one command removes the chance to stop when the number does not move the way you expect.
3. Touch nothing else. Not wal/, not chunks_head/, not the lock file, not any block directory. The WAL is the only copy of everything ingested since the last head compaction, and it is the one thing here whose deletion converts an outage into permanent loss.
4. Do not restart to reclaim anything. The restart has already been tried; on a full volume it frees nothing and costs twelve minutes of WAL replay during which even the history is unreadable.
5. Confirm ingestion resumes on its own once there is headroom, and record the first timestamp with fresh samples.
6. Fix the backup job today, because it runs again at 02:00. Add the removal of ${SNAP_ROOT}/${SNAP_NAME} after the upload succeeds, and make the job exit non-zero if that removal fails.
7. Repoint the disk alert at every mountpoint that matters, and add the absent() companion so an alert that stops matching becomes visible instead of becoming quiet.
8. Grow the volume last, if at all. It is a second brake: it buys time before the next occurrence and it also buys a longer delay before anyone notices one.
9. Write down the ingestion gap and where it will show up - notably the disagreement it creates between the local TSDB and anything remote-written to long-term storage for those hours.

Verification

1. Deletion frees space again. Watch prometheus_tsdb_time_retentions_total increment and confirm node_filesystem_avail_bytes rises with it. That counter incremented for sixty-four days while available bytes fell; one retention event that now returns bytes is the actual proof, and free space on its own is not.
2. The two accountings reconcile. Filesystem used bytes for the mountpoint should exceed prometheus_tsdb_storage_blocks_bytes by roughly the WAL plus the m-mapped head chunks and nothing more. A gap wider than that means inodes are still held.
3. The next backup run leaves nothing behind. Watch the run rather than reading its exit code: the tarball lands in S3 and data/snapshots is empty afterwards. The bug you are fixing is a job that reported success while accumulating state.
4. The disk alert selects something. node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"} must return data, and the rule evaluated against the last sixty-four days must show it would have fired weeks ago. A rule whose selector matches nothing is silent forever and looks exactly like a rule that is passing.
5. Ingestion is real, not merely up. prometheus_tsdb_head_samples_appended_total climbing and up at 1 across the estate, rather than a process that started and a readiness endpoint that answered.
6. The gap is documented from the data. First and last minute without samples, read out of Prometheus rather than off the incident timeline, published where people who query that window will find them.

Prevention

• Alert on the filesystem, per mountpoint, with a threshold rule and a predict_linear time-to-full rule, and add absent(node_filesystem_avail_bytes{mountpoint="..."}) beside them. Any selector naming a path, a device or a job can stop matching after ordinary infrastructure work, and this estate spent eighteen months proving it.
• Put the application’s storage gauge and df on the same panel and treat a widening gap between them as its own alertable condition. The gap is the signature of held bytes whatever is holding them.
• Know what --storage.tsdb.retention.size counts. It includes the WAL and the m-mapped head chunks in its total and deletes only persistent blocks, so it bounds the TSDB’s own footprint. It is not a bound on the volume and it will not save you from anything else living there.
• Treat anything that hard-links into a live data directory as a retention policy for that data. Snapshots, cp -al, rsync --link-dest, container layers: each one silently overrides the deletion policy of whatever it links, and none of it is visible in any application-level metric.
• Make every job that creates state responsible for removing it, with the removal inside the job’s success condition. A job measured only at the far end - does the object exist in S3? - cannot tell you it is filling the near end.
• Rehearse the restore, not just the backup. Sixty-four tarballs nobody has ever opened is sixty-four nights of disk and an untested assumption.
• Accept that the platform cannot alert on its own volume filling, because the rule engine is inside the process that stops. That specific condition needs an external check that pages independently of Prometheus.