Example: Storage Full

At 02:18 UTC Prometheus stops scraping. The /api/v1/query endpoint returns HTTP 503 with the message “storage unavailable”. The Grafana dashboards show a flatline: the last data point is 02:14 UTC. The on-call engineer connects to the Prometheus host and sees the disk at 100% utilisation; the data directory /var/lib/prometheus/data has consumed the volume. The page has been firing for four minutes without the on-call noticing, because the Prometheus alert rules were themselves unable to evaluate. This is the worked example.

What it is

A storage-full investigation is the six-phase loop applied to a disk-saturation event on the observability host. The investigation distinguishes between a real growth in series ingestion (legitimate pressure) and a retention misconfiguration (the platform is keeping too much) because the two have different mitigations.

A storage-full event is dangerous because the alert rules that should fire on it cannot evaluate. The platform is blind to itself at exactly the moment it needs to be most visible. The investigation must therefore start from the host, not from the metric.

Why a sysadmin cares

A storage-full event on the observability platform is a self-inflicted outage: the platform that is supposed to tell you about outages cannot tell you about itself. The mitigation must be both fast (the platform is blind) and correct (the wrong mitigation deletes the WAL and loses all unflushed data).

The worked example matters because storage-full events have unusual recovery semantics. Prometheus 2.55.x writes the WAL ahead of the head compaction; deleting the wrong file loses the last few hours of unflushed data. The mitigation must distinguish “extend the volume” from “prune the head blocks” from “delete the WAL”; the three are not interchangeable.

How it works

The investigation walks the six phases with the worked example running in parallel:

  02:14 UTC  Phase 1: symptom = "Prometheus scraping stopped; dashboards flatline after 02:14"
  02:22 UTC  Phase 2: impact  = "platform blind; cannot alert, cannot investigate, cannot diagnose"
  02:24 UTC  Phase 3: hypoth. = "disk full due to retention
                            misconfiguration; OR legitimate
                            growth from cardinality incident"
  02:28 UTC  Phase 4: evidence = (host disk, retention config,
                              ingestion rate, head series count)
  02:36 UTC  Phase 5: test    = retention misconfiguration
                            confirmed
  02:44 UTC  Phase 6: cause   = retention storage.tsdb.retention.
                            time set to 0 (forever);
                            mit. = set retention, restart, prune

Phase 1: Define the symptom

The on-call engineer cannot query Prometheus for the symptom; Prometheus is down. The symptom is observed at the host:

Prometheus scraping stopped at 02:14 UTC. Grafana dashboards show a flatline. Prometheus /api/v1/query returns 503 “storage unavailable”. Disk on /var/lib/prometheus is at 100%. The platform has been blind for 4 minutes.

The symptom is specific: time of failure, host, disk state, API response. The falsifier: when the API returns 200 and the disk is below 95%, the symptom resolves.

Phase 2: Quantify the impact

The impact is unusual: the platform that quantifies impact is broken. The on-call engineer estimates impact by inspection of the dashboards and the alertmanager state:

• Dashboards: flatline from 02:14 to current.
• Alertmanager: alerts that should be firing are not.
• On-call rotation: the platform that pages the on-call is silent.

# SEVERITY: READ-ONLY
# Confirm Prometheus is down.
curl -s -o /dev/null -w '%{http_code}' 'http://prometheus:9090/-/ready'

Expected output:

503

# SEVERITY: READ-ONLY
# Confirm the disk is at 100%.
df -h /var/lib/prometheus

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3  500G  500G     0 100% /var/lib/prometheus

Phase 3: Form a hypothesis

Two hypotheses are plausible:

H1 (retention misconfiguration). The retention setting storage.tsdb.retention.time was changed to 0s (forever) by a recent config change. Prometheus is keeping all blocks forever; the disk has filled. The fix is to set the retention to the documented value (30 days) and restart.

H2 (legitimate growth). A cardinality incident has produced an unexpected volume of series; even with the configured 30-day retention, the disk cannot keep up. The fix is to extend the volume and address the cardinality incident.

The hypotheses have different falsifiers:

F1. If the retention setting in /etc/prometheus/prometheus.yml is 0s and the head series count is stable, H1 is confirmed. If the retention is 30 days and the head series count has doubled in the last 24 hours, H2 is more likely.

F2. If the disk filled because the volume was sized for the previous 30-day retention and the configured retention is unchanged, H2 is more likely. If the disk filled because the retention was changed, H1 is more likely.

Phase 4: Find evidence

Four surfaces, all at the host:

1. Host disk. The disk is at 100%; the data directory has consumed the volume. Confirm the breakdown:

# SEVERITY: READ-ONLY
du -sh /var/lib/prometheus/data/* | sort -h

Expected output:

4.0K    /var/lib/prometheus/data/lock
12K     /var/lib/prometheus/data/wal
240G    /var/lib/prometheus/data/chunks_head
260G    /var/lib/prometheus/data/blocks

The blocks directory is 260 GB and the chunks_head is 240 GB. Total 500 GB, which matches the volume size. The distribution is consistent with both hypotheses; the retention setting is the next evidence.

2. Retention configuration. The retention setting:

# SEVERITY: READ-ONLY
grep -E 'retention' /etc/prometheus/prometheus.yml

Expected output:

  storage.tsdb.retention.time: 0s

Retention is 0s (forever). H1 is consistent. The recent config change is in version control:

# SEVERITY: READ-ONLY
cd /etc/prometheus && git log --oneline -5 prometheus.yml

Expected output:

a8f3c2d 2026-08-12  set retention to 0s for audit retention
7d1e9b4 2026-08-10  bump scrape interval to 15s
4f0a2c1 2026-08-08  initial

The change at a8f3c2d set retention to 0s. The commit message says “for audit retention”. H1 is confirmed.

3. Ingestion rate. The Prometheus ingestion rate over the last 24 hours, observed via the head series count in the chunks_head directory:

# SEVERITY: READ-ONLY
ls /var/lib/prometheus/data/chunks_head/ | wc -l

Expected output:

14832

14,832 head series. A typical Prometheus on this platform holds 12,000-15,000 head series; the count is within band. H2 is refuted by the head series count (the cardinality has not changed materially).

4. Block count. The number of compacted blocks:

# SEVERITY: READ-ONLY
ls /var/lib/prometheus/data/blocks/ | wc -l

Expected output:

730

730 blocks at 360 MB per block = ~260 GB. With 30-day retention, the platform holds approximately 30 blocks per day x 30 days = 900 blocks. With retention at 0s, the platform holds every block ever compacted, which at 1 block every 2 hours since the platform was deployed 60 days ago would be 720 blocks. The count is consistent with retention at 0s and inconsistent with retention at 30 days.

Phase 5: Test the hypothesis

H1 predicted: retention is 0s; the block count is consistent with no deletion; the head series count is unchanged. The config, the commit history, the block count, and the head series count all confirm. H1 is confirmed; H2 is refuted.

Phase 6: Locate root cause and document

Root cause: the config change at a8f3c2d set storage.tsdb.retention.time to 0s for “audit retention”. The intent was to keep all blocks for a future audit; the effect was that the disk filled within 30 days because the volume was sized for 30-day retention.

Mitigation: extend the volume (or move the data directory to a larger volume), then set the retention to the documented value (30 days) and restart Prometheus. The restart causes Prometheus to delete blocks older than 30 days on the next compaction cycle.

Runbook log entry, written at 02:50 UTC:

# Runbook: ObservabilityStorageFull

## Symptom (Phase 1)
Prometheus scraping stopped at 02:14 UTC. Grafana dashboards
flatline from 02:14. /api/v1/query returns 503 "storage
unavailable". Disk on /var/lib/prometheus at 100%.

## Impact (Phase 2)
Platform blind for 4+ minutes at time of detection. Cannot
alert; cannot investigate other incidents from this platform.
Recovery required before platform can resume normal function.

## Hypothesis (Phase 3)
H1: storage.tsdb.retention.time was changed to 0s by recent
config commit. Disk fills because volume sized for 30 days.
H2: legitimate cardinality growth exceeding volume capacity.
Falsifier F1: if retention is 0s and head series count is
stable, H1 confirmed. If retention is 30 days and head series
count doubled, H2 more likely.
Falsifier F2: if volume is sized for previous retention and
configured retention is unchanged, H2 more likely. If volume
is sized for previous retention and configured retention was
changed, H1.

## Evidence (Phase 4)
- disk: /var/lib/prometheus at 100% (500G used of 500G).
- directory breakdown: blocks 260G, chunks_head 240G, wal 12K.
- config: storage.tsdb.retention.time: 0s in prometheus.yml.
- git log: commit a8f3c2d at 2026-08-12 set retention to 0s
  with message "for audit retention".
- head series count: 14,832 (within 12k-15k band).
- block count: 730 (consistent with no deletion; inconsistent
  with 30-day retention).

## Test (Phase 5)
H1 confirmed. H2 refuted (head series count is within band;
the growth is retention-driven, not cardinality-driven).

## Root cause + mitigation (Phase 6)
Root cause: retention setting 0s in commit a8f3c2d. Volume
sized for 30-day retention.
Mitigation:
  1. Extend volume to 750G (or move data dir to larger vol).
  2. Revert retention to 30d in prometheus.yml.
  3. Restart Prometheus.
  4. Wait for next compaction cycle (~2 hours) to prune
     blocks older than 30 days.
  5. Verify disk usage drops below 60% within 4 hours.
Recovery completed at 02:48 UTC; disk at 100% at start of
recovery, 100% immediately after volume extend (no pruning
yet), 58% at 06:48 UTC after two compaction cycles.

## Follow-up
- Document retention setting in deploy runbook: 30 days is
  the documented value; deviation requires volume resize.
- Add pre-deploy check: retention time must be one of the
  documented values (1d, 7d, 30d, 90d).
- Add capacity dashboard panel: predicted time-to-fill based
  on current ingestion rate.
- Add alert at 80% disk usage (severity: ticket); page at
  95% (severity: page).
- Add self-monitoring rule: scrape timeout alert on
  Prometheus itself (Prometheus scrapes its own /-/ready).

How to configure it

The investigation produces three configuration artefacts. The immediate artefact is the volume extension and the retention revert. The follow-up artefacts are a pre-deploy check, a self-monitoring rule, and an alert at 80% / 95%.

A pre-deploy check that enforces the retention invariant:

# deploy-checks/prometheus-retention.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: prometheus-retention-check
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: check
        image: curlimages/curl:8
        command:
        - sh
        - -c
        - |
          set -eu
          # READ-ONLY: verify retention setting is one of
          # the documented values.
          RETENTION=$(curl -s \
            'http://prometheus:9090/api/v1/status/config' \
            | jq -r '.data.yaml | capture("storage.tsdb.retention.time: ?(?<v>\\S+)").v')
          case "$RETENTION" in
            1d|7d|30d|90d)
              echo "OK: retention $RETENTION is documented."
              ;;
            0s)
              echo "FAIL: retention 0s requires explicit volume resize."
              exit 1
              ;;
            *)
              echo "FAIL: retention $RETENTION is not documented."
              exit 1
              ;;
          esac

A recording rule that predicts time-to-fill:

groups:
- name: prometheus-host.rules
  interval: 30s
  rules:
  - record: prometheus:disk_usage:predicted_fill_time_hours
    expr: |
      (node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"}
        / 1024 / 1024 / 1024)
      / (rate(node_filesystem_writes_bytes{mountpoint="/var/lib/prometheus"}[1h])
        / 3600)

An alert at 95% disk usage:

  - alert: PrometheusStorageCritical
    expr: |
      (node_filesystem_size_bytes{mountpoint="/var/lib/prometheus"}
        - node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"})
      / node_filesystem_size_bytes{mountpoint="/var/lib/prometheus"} > 0.95
    for: 5m
    labels:
      severity: page
      team: observability
      service: prometheus
    annotations:
      summary: 'Prometheus disk usage above 95% for 5 minutes'
      runbook_url: 'https://runbooks.example.com/prometheus/storage-full'

How to validate it

Validate that the investigation reached phase 6 by replaying each step against the live platform:

# SEVERITY: READ-ONLY
# 1. Confirm Prometheus is back in steady state.
curl -s -o /dev/null -w '%{http_code}' 'http://prometheus:9090/-/ready'

Expected output:

200

# SEVERITY: READ-ONLY
# 2. Confirm the retention setting is reverted.
grep -E 'retention' /etc/prometheus/prometheus.yml

Expected output:

  storage.tsdb.retention.time: 30d

# SEVERITY: READ-ONLY
# 3. Confirm the disk is recovering.
df -h /var/lib/prometheus

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3  750G  440G  310G  59% /var/lib/prometheus

# SEVERITY: READ-ONLY
# 4. Confirm the block count is dropping toward 30-day retention.
ls /var/lib/prometheus/data/blocks/ | wc -l

Expected output:

360

The four checks confirm: the symptom has resolved (/ready returns 200), the cause has been removed (retention is 30d), the recovery is progressing (disk at 59%, blocks at 360), and the follow-up signal is live (alert rule loaded).

How it can fail

Five specific failure shapes for a storage-full investigation:

1. Deleting the WAL. The on-call sees the data directory is full, finds the WAL at 12 KB, and deletes it to free space. The WAL contains all unflushed data; deleting it loses the last few hours of metrics. The platform recovers but with a gap. Symptom: the gap appears as a flatline in Grafana for the WAL age; the on-call did not realise the WAL was the unflushed buffer.

2. Pruning head blocks without understanding the compaction cycle. The on-call finds chunks_head at 240 GB and deletes the directory. The head block is the in-memory buffer; deleting it forces a full re-ingestion of all series. The platform recovers but at much higher memory cost and with a long warm-up. Symptom: memory spikes to 90% on restart; scrape latency rises for 30 minutes.

3. Extending the volume without reverting retention. The on-call extends the volume from 500 GB to 750 GB but leaves retention at 0s. The platform is no longer full but is still keeping every block forever. The next fill event is in 30 days; the same incident recurs. Symptom: the immediate symptom resolves; the same symptom recurs at the next 30-day boundary.

4. Restarting Prometheus without addressing the root cause. The on-call restarts Prometheus to free memory; the retention setting is unchanged; the head block re-fills the disk within hours. Symptom: the immediate symptom resolves; the platform fills again before the on-call signs off the incident.

5. Failing to add the self-monitoring alert. The incident is closed without a self-monitoring rule. The same incident recurs in six months when a different config change fills the disk; the platform is blind again. Symptom: the runbook log entry names the follow-up but the alert is not in the rules file.

How to troubleshoot it

When the storage-full investigation is taking longer than expected, the diagnostic order is:

1. Confirm the symptom is still observable. Is the disk still at 100%? Has the platform restarted? If the platform has restarted, the symptom may have resolved and the investigation may not need to continue.
2. Read the retention setting first. The retention setting is the cheapest evidence; it is in a single file and can be read in seconds. Do not open Grafana first; Grafana is blind.
3. Count the blocks. The block count is the second cheapest evidence; it is in a single directory and can be counted with ls | wc -l. The block count names the retention regime.
4. Check the head series count. The head series count is the third cheapest evidence; it is the discriminator between H1 and H2.
5. Mitigate before complete the investigation if the platform is needed. The platform is blind. If another incident is in progress, the on-call may need to restore the platform before the storage-full investigation completes. Extend the volume first; investigate second.

Security implications

The recovery procedure extends the volume and reverts the retention setting. The retention setting affects how long metrics are kept; in some jurisdictions, longer retention means more data is subject to discovery in a legal hold. The audit-driven intent of the original change (“for audit retention”) may have been correct; the implementation (set to 0s without volume resize) was the failure.

Document the retention setting in the data governance policy. A retention setting that violates the data governance policy is a compliance issue, not just an operational issue.

Performance implications

A storage-full event causes Prometheus to refuse writes. The refusal is fast (the WAL write fails; the scrape returns 500). The recovery (restart, compaction cycle) is slower: the compactor must walk all blocks older than the new retention (30 days) and delete them. On a 500 GB dataset this takes 30 minutes to 2 hours.

The volume extension is the longest operation; it depends on the storage backend. A local disk extension takes seconds (growpart, resize2fs). A cloud volume extension takes minutes (AWS EBS, GCP persistent disk). A network volume (NFS, Ceph RBD) takes longer.

The follow-up recording rule evaluates every 30 seconds against the disk usage counter; the cost is negligible.

Production guidance

• The disk size and the retention setting are coupled. A change to one without the other is the failure shape. Document the coupling in the deploy runbook.
• The self-monitoring rule is non-optional. A Prometheus instance that does not scrape itself cannot alert on its own failure.
• The 80% / 95% alert pair is the standard. 80% tickets; 95% pages. The 80% ticket gives the working-hours team time to resize the volume before the 95% page.
• The volume resize is reversible. The data deletion is not. Resize first; investigate the root cause second.
• The retention setting is a deployment invariant. The pre-deploy check enforces it; the runbook documents it.
• Coordinate with the data governance team. A retention setting that violates the governance policy is a compliance issue.

Verification

You should now be able to answer:

• Why does the storage-full investigation start from the host rather than from the metric?
• Which filesystem artefact is the cheapest falsifier between a retention hypothesis and a cardinality hypothesis, and what does it show?
• Why is “extend the volume” the right first mitigation and “delete the WAL” the wrong one?
• What is the role of the self-monitoring rule, and why is it non-optional?
• Why is the retention setting a deployment invariant, and what enforces it?

Quiz