high riskcluster affecting~60 min

Runbook: Observability Storage Full

1 · Prerequisites

Confirm every item is in place before any state change.

Example: Storage Full
Storage Monitoring
Retention Basics
Loki Retention Configuration
Tempo Retention
The Cardinality Budget
Root or sudo on the affected host, and the ability to extend the volume without a support ticket
The configured retention values for every surface, and the provisioned capacity each one was sized against

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Start from the host, not from a query. This is the one incident class where the alerting that should have warned you cannot evaluate, because rule evaluation is a write and the volume is full. df -h and df -i are the first two commands, and the dashboard is not evidence about anything right now
· Identify which surface is actually full. The platform has several and they fail differently: the Prometheus TSDB volume, the write-ahead log on that same volume, the Loki or Tempo compactor working directory, and the object store, which does not fill so much as throttle and bill. Treating one as another produces a fix aimed at the wrong component
· Check inodes as well as bytes. The write-ahead log is a sequence of segment files, and a high-ingest instance can exhaust inodes on a filesystem that still reports free space. df -h looks fine, writes fail anyway, and the symptom is identical
· Establish whether this is retention or growth before any deletion. Retention means the platform is keeping more than it was configured to; growth means it is receiving more than it was sized for. The two have opposite fixes and the cheapest falsifiers — block count and head series count — can both be read with ls while Prometheus is down
· Confirm what the configured retention actually is, from the flags the process is running with rather than from what somebody remembers. Prometheus retention is a command-line flag, not a prometheus.yml key, and reading the wrong file is how a team concludes retention is set when it is not
· Establish whether the last blocks are evidence for an incident in progress. Freeing space by deleting the oldest data is usually fine; freeing it by deleting the newest is deleting the record of what you are currently investigating
· Know before you change any retention value that the change is enforced by a compactor, is not instantaneous, and is not reversible. It will not free space in time to save you today, and when it does run it deletes permanently
· Have the volume-extension path confirmed as available before the deletion path is considered. If storage can be added in ten minutes, nothing in this runbook should delete anything

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the host first: df -h and df -i on the affected mount, then du -sh across the data directory to see which component owns the space. The platform cannot tell you this about itself right now
2Classify the surface — Prometheus TSDB volume, the write-ahead log on it, a component working directory, or the object store — and say which one out loud. Every subsequent step differs, and the object store in particular is a cost and throttling problem rather than a disk problem
3Buy time with the safe lever before diagnosing anything further. Extending the volume or moving the data directory to a larger one is non-destructive, reversible in practice, and restores the platform's function immediately. It is always the correct first move when it is available
4Only if extension is genuinely unavailable, prune the oldest compacted blocks — never the write-ahead log, and never the head. That ladder is graded by permanence: extend the volume is safe, drop the oldest blocks is lossy but bounded and known, delete the write-ahead log discards every sample not yet compacted and cannot be undone
5Restore function before you restore history. Getting the process writing again is what stops the platform being blind; recovering the space efficiently is a slower job that does not block paging
6Now separate retention from growth. Count the compacted blocks and read the head series count: a block count far above what the configured retention implies is a retention failure, while a head series count well above its normal band is growth, usually a cardinality event
7If it is retention, fix the retention flags — and set the size cap as well as the time cap. The size cap is the safety belt that drops the oldest blocks before the volume fills; without it Prometheus writes until the disk is full and crashes on the write
8If it is growth, do not answer it with deletion. Deleting history to make room for an unintended metric explosion buys hours and loses evidence; the fix is upstream, at the exporter, the relabelling, or the instrumentation that started producing it
9For Loki, remember that the effective retention for any stream is the longest of the global, per-tenant and stream-label values. Shortening the global value changes nothing for a stream whose own label sets a longer one, and the storage keeps growing while the change is believed to have worked
10For Tempo, confirm the compactor is actually running before concluding that retention is misconfigured. The compactor is the only component that deletes; with it stopped, retention is theoretical and the bucket grows regardless of the configured value
11Leave headroom deliberately when sizing. Compaction needs free space to write a merged block before it removes the originals, so a volume sized exactly to the data is a volume that cannot compact
12Add or repair the projection alert before closing the incident. An alert on time-to-exhaustion catches this days ahead; an alert on a percentage threshold fires at the same number for a disk with six days left and one with a hundred
13Record what actually filled it, the artefact of the fix, and the data lost. If blocks were pruned, name the time range that is now missing so the next investigator can tell a gap from a fault

4 · Verification

Confirm the procedure actually fixed the problem.

✓df -h shows sustained free space on the affected mount, and df -i shows free inodes. Both, not one: a volume with space and no inodes fails writes exactly the same way
✓The free space is still free ten minutes later. Space reclaimed by a fix that has not addressed the cause is space that is about to be consumed again, and the second fill happens faster because it starts from a compaction backlog
✓Prometheus answers /-/ready, count(up == 1) is close to the expected target count, and /api/v1/rules shows the expected groups with recent evaluation timestamps. Scraping resumed is not the same as alerting resumed
✓promtool tsdb list against the data directory reports a block count consistent with the configured retention, rather than every block since the instance was deployed
✓The retention flags the process is actually running with are the ones intended. Read them from the running process, not from the file somebody edited, and confirm both the time cap and the size cap are present
✓For Loki, the effective retention has been checked against all three rules — global, per-tenant, and stream label — not just the global one that was edited
✓For Tempo, the compactor is running and its last successful compaction is recent. A configured block_retention with no compactor deletes nothing
✓The object-store side shows no sustained throttling or upload backlog. A component that cannot flush to the bucket fills its local working directory next, which is the same incident arriving from a different direction
✓A time-to-exhaustion projection alert exists for the surface that filled, is firing on test data, and routes somewhere a human reads
✓The data gap is declared with times if anything was pruned: which range of metrics, logs or traces is now permanently absent

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Extending a volume is the only step here with a clean rollback, and it is rarely worth rolling back. Everything else on the deletion side is irreversible by construction — the blocks are gone, the write-ahead log is gone, the chunks are gone
↶A retention value that was lowered can be raised again, but the data deleted while it was low does not come back. Treat a retention reduction as a deletion with a delay, because that is exactly what it is
↶If a retention change was applied and the compactor has not yet run, reverting it before the sweep is the one genuine undo available. That window is short and it is measured by the compaction interval, so check the interval before assuming there is time
↶Do not restart a component repeatedly to clear space. A restart on Prometheus replays the write-ahead log before the instance serves anything, so a loop of restarts on a full volume produces a long blind window and no free space
↶Do not delete the write-ahead log to buy room. It holds every sample not yet compacted into a block, the loss is silent, and the resulting gap is in the most recent data, which is the data the current incident needs
↶If the fix was a temporary volume or a moved data directory, that is drift until it exists in the configuration that builds the host. Record it as a follow-up with an owner, or the next rebuild reproduces the original sizing

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the storage or platform owner before deleting anything. Deletion is the irreversible branch of this runbook, and if extending the volume is available at all it is preferable to every alternative below it
· Escalate to whoever owns the ingest source if the cause is growth rather than retention. A cardinality event or a log-volume spike is fixed at the producer; the storage team can only absorb it, and absorbing it is a bill rather than a solution
· Escalate to the compliance or data owner before shortening any retention that covers regulated telemetry. A retention reduction is a deletion, and a deletion of audit-relevant logs is not an operational decision
· Escalate to the object-store or cloud owner if the surface is a bucket rather than a disk. Quotas, lifecycle policies and request throttling are theirs, and the symptoms reach you as component failures that look local
· Escalate to the vendor immediately if the affected tier is managed, before attempting anything self-service. A managed backend has capacity levers that are not exposed to you
· Escalate if the platform has been blind long enough that other incidents may have gone undetected. That window is a fact other teams need, not an internal detail of this repair

Storage exhaustion on the observability platform is the incident class with no warning shot. Rule evaluation is a write. When the volume is full, the rules that would have paged somebody about a full volume cannot evaluate, so the first anyone hears about it is a dashboard that stops updating — and by then the platform has been blind for however long it took a human to notice a flat line.

That has two consequences for how this runbook is written. Every diagnostic starts at the host, because the platform cannot answer questions about itself. And the first action is not diagnosis, it is buying back the capacity to write, using the one lever that cannot make anything permanently worse.

When this runbook applies

A volume holding the Prometheus TSDB, a component working directory, or a compactor scratch path is at or near full.
A component has crashed on a write error, or is refusing writes.
Ingestion has stopped and the dashboards have flatlined with no corresponding change in production.
The object store behind Loki or Tempo is throttling, over quota, or growing without a ceiling.

When it does not

The disk is filling but has not filled. That is capacity planning, and it is a much better conversation to have. Use the projection alert and fix it during working hours.
Queries are slow but writes are fine. That is a query or compaction performance problem, not exhaustion, and shrinking retention to fix it deletes data for no reason.
The bill grew, nothing broke. Object stores do not fill, they charge. That is a retention and tiering decision, made deliberately, not an incident response.

Which surface is full?

They fail differently and the fixes do not transfer.

Surface	How it fails	What it looks like
Prometheus TSDB volume	Writes fail, the process crashes	Scraping stops, dashboards flatline, the API returns errors
Write-ahead log, on the same volume	A partial segment cannot be reused; writes stop	Identical to the above, and may be an inode failure rather than a byte failure
Component working directory	Flushes and compaction fail	Ingestion backs up, then drops
Object store	Throttling, quota, or cost	Upload backlog, 429 responses, a bill nobody approved

The two that are most often confused are the last two. A component that cannot flush to the object store fills its own local working directory next, so an object-store problem arrives at the on-call engineer as a local disk problem on a different host entirely.

Read-only / Safestart at the host, because the platform cannot answer

# Substitute your own values before running:
TSDB=/var/lib/prometheus/data

# Bytes and inodes. A volume with space and no inodes fails writes
# exactly the same way, and reports itself as healthy.
df -h "$TSDB"
df -i "$TSDB"

# Who owns the space?
du -sh "$TSDB"/* | sort -h

# The retention Prometheus is ACTUALLY running with. It is a flag,
# not a prometheus.yml key, so read the process, not the file.
ps -o args= -C prometheus | tr ' ' '\n' | grep -- '--storage.tsdb'

That last command is worth its place. Prometheus retention is set by --storage.tsdb.retention.time and --storage.tsdb.retention.size on the command line; there is no retention key in prometheus.yml. A team that has written one into the YAML has a retention setting that does nothing, and the file they keep re-reading during the incident agrees with them.

Step 1 — buy time with the safe lever

The three levers are graded by permanence. Never invert the order.

Extend the volume, or move the data directory to a larger one. Safe. Nothing is deleted, the platform resumes writing, and every diagnosis below becomes something you do calmly instead of at speed. If this is available at all, it is the correct first action and nothing else in this section should happen.

Prune the oldest compacted blocks. Lossy, but bounded and knowable — you can say exactly which time range you gave up, and it is the range least likely to matter to the incident in front of you.

Delete the write-ahead log. Catastrophic, and it is the option that looks most attractive at 02:30 because the directory is right there. It holds every sample not yet compacted into a block. Deleting it silently discards the most recent data — the data about what is happening now — and there is no recovery.

The pressure during this incident pushes exactly the wrong way: the safe lever needs someone with authority over storage, and the catastrophic one needs a single command. Extend the volume, and page whoever can approve it rather than working around them.

Restoring function comes before restoring efficiency. The platform being able to write again is what ends the blindness; reclaiming space elegantly is a slower job that does not block paging and should not be allowed to.

Note also that a restart is not a way to clear space. Prometheus replays the write-ahead log before it serves a query or evaluates a rule, so a loop of restarts against a full volume produces a long additional blind window and frees nothing.

Step 2 — retention or growth?

They present identically on a full disk and their fixes are opposites. Retention failure means the platform is keeping more than it was configured to keep. Growth means it is receiving more than it was sized for. Deleting data is a reasonable response to the first and an actively harmful response to the second.

Both falsifiers can be read with the process down, which matters, because the process is frequently down.

Read-only / Safetwo counts that separate the hypotheses

# Substitute your own values before running:
TSDB=/var/lib/prometheus/data

# How many compacted blocks exist? Prometheus compacts the head every
# two hours, so the count implied by a retention window is arithmetic.
# Far more blocks than the window implies means retention is not being
# enforced.
promtool tsdb list "$TSDB"
promtool tsdb list "$TSDB" | wc -l

# Is the series count normal, or has it grown? Compare against the band
# this instance normally sits in, which should be in the capacity notes.
du -sh "$TSDB"/wal
ls "$TSDB"/chunks_head | wc -l

If the block count is far above what the configured window implies and the series count sits in its usual band, this is retention: the platform is keeping everything and the volume was sized for a window. If the series count has jumped and retention looks correct, this is growth, and it is usually a cardinality event with a recent cause.

Step 3 — fix retention properly, and understand the delay

Two caps, not one. The time cap expresses the policy; the size cap is the safety belt that keeps the volume from filling when the policy turns out to be wrong.

Configuration changeretention lives in the service flags

# Substitute your own values before running:
# /etc/default/prometheus - retention is set by flag; prometheus.yml
# has no key for it. Size the cap BELOW the volume, with headroom:
# compaction has to write a merged block before it removes the originals,
# so a volume sized exactly to the data cannot compact.
ARGS="--storage.tsdb.path=/var/lib/prometheus/data \
    --storage.tsdb.retention.time=30d \
    --storage.tsdb.retention.size=200GB"

The size cap drops the oldest blocks once the limit is reached. It does not stop writes and it is not a quota — it is the difference between Prometheus shedding history and Prometheus crashing on a write error.

A retention change is a delayed, permanent deletion — and it will not save you today

Nothing about a retention change is immediate. Prometheus removes blocks on its next compaction cycle. Loki’s compactor sweeps on its own interval and honours a delete delay after that. Tempo’s compactor runs on an interval too, and if it is not running at all then retention deletes nothing regardless of what the configuration says.

So a retention change is not a mitigation for a disk that is full right now — that is what step one is for. It is prevention, applied after the fire is out.

And when it does run, it deletes permanently and all at once. Lowering a retention value, or enabling retention on a store that has been keeping everything, hands the compactor a backlog of everything already past the new horizon. That sweep is not a gradual trim; it is a single bulk deletion of every object older than the new value, and nothing brings it back.

Step 4 — Loki and Tempo have their own traps

Loki’s effective retention is the longest of three rules, not the one you edited. A global value, a per-tenant override, and a stream-label value all apply, and the longest wins. Shortening the global while an audit stream carries a longer label changes nothing for that stream, the storage keeps growing, and everybody believes the fix landed.

Tempo deletes only through its compactor. The retention value is compactor.compaction.block_retention and it defaults to keeping blocks forever. A cluster where the compactor has stopped — after an upgrade, after a lock conflict, after a restart nobody noticed — grows without limit while its configuration says ninety days. Before concluding that retention is misconfigured, confirm the process that enforces it is running and has swept recently.

Both are also worth checking from the object-store side, because a component that cannot write to the bucket fills its local disk instead.

Read-only / Safeis the bucket path healthy, or is the backlog local

# Substitute your own values before running:
PROM=http://prometheus.example.com:9090

# Object-store requests being throttled. 429s here become an upload
# backlog, and the backlog becomes a full local volume.
curl -s -G "$PROM/api/v1/query" --data-urlencode \
'query=sum by (status_code) (rate(loki_objectstore_request_duration_seconds_count{status_code=~"4..|5.."}[5m]))' \
| jq -r '.data.result[] | [.metric.status_code, .value[1]] | @tsv'

# Free bytes on the affected mount, as the platform sees it.
curl -s -G "$PROM/api/v1/query" --data-urlencode \
'query=node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"}' \
| jq -r '.data.result[0].value[1] // "no data"'

Step 5 — fix the alert that should have caught this

A percentage threshold is the wrong shape for this failure. A volume at 90 percent that grows a tenth of a percent a day and a volume at 50 percent that grows five percent a day produce the same page at different times and the wrong one is urgent.

The right alert is a projection: how long until this surface is full at the current rate. predict_linear over the available-bytes series gives exactly that, and it fires days ahead rather than minutes after.

Add the inode check alongside it. Inode exhaustion produces identical symptoms on a filesystem that reports free space, and no byte-based alert will ever see it coming.

Common patterns

Symptom	Likely cause	Resolution
Scraping stopped, dashboards flatline, no production change	Volume full; rule evaluation cannot write, so nothing paged	Start at the host: `df -h`, `df -i`, `du -sh`
Disk shows free space, writes still fail	Inode exhaustion from write-ahead-log segments	`df -i`; size the filesystem for file count, not only bytes
Block count far exceeds the retention window	Retention not enforced, or set only in `prometheus.yml` where it does nothing	Read the flags from the running process; set both time and size caps
Series count jumped, retention correct	Growth, usually a cardinality event	Fix the producer; deletion only buys time
Retention shortened, storage unchanged	Loki: a longer per-tenant or stream rule wins. Tempo: the compactor is not running	Check all three Loki rules; confirm the Tempo compactor swept recently
Space freed, gone again within a day	The cause was never addressed; the refill starts from a compaction backlog	Return to step two before closing
Recent data missing after the fix	The write-ahead log was deleted to buy room	Nothing. Declare the gap and record how it happened
Compaction failing on a volume with free space	Not enough headroom to write a merged block before removing the originals	Size the cap below the volume, deliberately

Escalation

Escalate when:

Before deleting anything, to whoever can extend the volume. Deletion is the irreversible branch and extension usually is not.
The cause is growth. The producer’s owner has the only real fix.
Any retention covering regulated telemetry is about to be shortened. That is a deletion decision, not an operational one.
The surface is a bucket. Quotas, lifecycle rules and throttling belong to the storage owner and reach you disguised as local failures.
The tier is managed. Open the vendor case before trying anything.
The platform has been blind long enough that other incidents may have gone undetected. Other teams need that window.

Why this matters

Two things make this incident different from every other one in the course, and both are about information rather than storage.

The first is that the platform cannot warn you about it. Rule evaluation needs to write, so the failure suppresses its own alert, and the detection path is a human noticing that a graph stopped. That is why the projection alert matters so much more here than a threshold does: it has to fire days early, while there is still a working platform to fire it.

The second is that the fastest fix and the correct fix point in opposite directions. Deleting something is one command and needs nobody’s approval; extending a volume needs a person who owns storage and may be asleep. So the pressure of the incident pushes toward the irreversible option at exactly the moment when judgement is worst — and the thing being deleted, if you reach for the write-ahead log, is the record of the last few hours, which is the period you are going to be asked about.

Decide the ladder before the incident. Extend, then prune the oldest, and never the log.