Storage Monitoring

The disk that holds the Prometheus TSDB fills up at 04:00. The Prometheus process crashes on a write error. The WAL replay on restart takes forty minutes. Every dashboard in the company is empty for an hour. The cause is a retention setting that was not enforced, and a disk-provisioning plan that did not account for the actual growth rate.

Storage exhaustion is the single most common production failure in a Prometheus stack. The platform has four storage surfaces; each one has its own exhaustion mode; each one needs its own metric and its own alert.

What it is

The Prometheus stack has four storage surfaces:

1. Prometheus TSDB on local disk. The active TSDB directory (--storage.tsdb.path) holds the head block and the persisted blocks.
2. Prometheus WAL on local disk. The write-ahead log lives in the wal/ subdirectory of --storage.tsdb.path and holds the segment files that buffer in-flight samples before they are compacted into blocks. There is no separate flag for it; the only way to put the WAL on a different device is to mount that device at wal/.
3. Object store bucket. Loki, Tempo, Mimir, and Cortex all use an object store (S3, GCS, Azure Blob, MinIO) for long-term retention of chunks, blocks, or trace data.
4. Loki ingester memory + boltdb-shipper. Loki’s ingester holds active streams in memory and flushes them to the object store via the boltdb-shipper.

The right approach is to monitor each surface with its own metric and to alert on a time-to-exhaustion projection, not only an absolute threshold.

Why a sysadmin cares

The failure modes for the four surfaces are different but the operational consequence is the same: data loss or visibility loss. A Prometheus with a full disk loses every alert that fires during the outage and replays the WAL slowly on recovery. A Loki ingester with no memory for new streams silently drops logs. A Tempo ingester that cannot flush to the object store starts rejecting traces.

The right self-monitoring projects when the surface will fill, not whether it is full today. A 50 percent full disk that is filling at 5 percent per day is a six-day page; a 90 percent full disk that is filling at 0.1 percent per day is a 100-day page. Both have the same percentage; only the projection says which one is urgent.

How it works

+----------------------------------------------+
| Prometheus host                              |
|   /var/lib/prometheus/                       |
|     - chunks_head/         (head block)      |
|     - wal/                 (WAL segments)    |
|     - blocks/              (compacted blocks)|
|     - queries.active                       |
+--------------------+-------------------------+
                     |
                     v
+----------------------------------------------+
| node_exporter                                 |
|   node_filesystem_avail_bytes{mount="/var"}   |
|   node_filesystem_files_free{mount="/var"}    |
|   node_disk_written_bytes_total{device="sda"} |
+--------------------+-------------------------+
                     |
                     v
+----------------------------------------------+
| Loki / Tempo / Mimir                          |
|   ingester -> boltdb-shipper -> S3 bucket    |
|   +-------------------+----------------------+
|                       v
+----------------------------------------------+
| Object store                                  |
|   - request count, 4xx, 5xx per request       |
|   - throttle count                            |
|   - bucket size (BytesTotal)                  |
+----------------------------------------------+

Each surface has a metric:

• TSDB disk: prometheus_tsdb_storage_blocks_bytes
• TSDB head: prometheus_tsdb_head_series
• Object store (via Loki): loki_ingester_streams, loki_boltdb_shipper_uploader, loki_objectstore_request_duration_seconds
• Local disk (via node_exporter): node_filesystem_avail_bytes, node_filesystem_files_free

The time-to-exhaustion projection is a PromQL predict_linear over the bytes-used series. It returns “seconds until the series crosses the threshold”.

Under the hood

How to configure it

Disk-fill projection alerts

# /etc/observer/rules/storage.yml
groups:
  - name: storage-disk
    rules:
      # 1. Prometheus TSDB disk fills in 7 days at current rate.
      #    The projection uses predict_linear on the bytes-used
      #    series, projected to the filesystem size threshold.
      - alert: PrometheusDiskWillFillSoon
        expr: |
          predict_linear(
            node_filesystem_avail_bytes{
              mountpoint="/var/lib/prometheus"
            }[6h],
            7 * 24 * 3600
          ) < 0
        for: 1h
        labels:
          severity: critical
          team: platform
        annotations:
          summary: |
            Prometheus disk will be full in 7 days at current
            usage rate. Investigate retention or provision more
            capacity.

      # 2. WAL storage size above a known ceiling.
      #    Catches inode-style exhaustion.
      - alert: PrometheusWALStorageHigh
        expr: |
          prometheus_tsdb_wal_storage_size_bytes
            > (50 * 1024 * 1024 * 1024)
            # 50 GiB - tune to the actual WAL throughput
        for: 30m
        labels:
          severity: warning
          team: platform }

      # 3. Filesystem inodes free below a ceiling.
      #    Catches the inode-exhaustion failure shape.
      - alert: PrometheusInodesLow
        expr: |
          node_filesystem_files_free{
            mountpoint="/var/lib/prometheus"
          } < 100000
        for: 10m
        labels:
          severity: warning
          team: platform

      # 4. TSDB storage blocks above a known ceiling.
      #    Catches the case where retention is not enforced
      #    and blocks keep accumulating.
      - alert: PrometheusStorageBlocksHigh
        expr: |
          prometheus_tsdb_storage_blocks_bytes
            > (500 * 1024 * 1024 * 1024)
            # 500 GiB - tune to provisioned capacity
        for: 30m
        labels:
          severity: warning
          team: platform

  - name: object-store
    rules:
      # 5. Object store request failures (Loki/Tempo/Mimir).
      - alert: ObjectStoreRequestFailing
        expr: |
          sum by (op) (
            rate(loki_objectstore_request_duration_seconds_count{
              status_code=~"5.."
            }[5m])
          ) > 0
        for: 10m
        labels:
          severity: warning
          team: platform }

      # 6. Object store request throttling.
      #    Loki/Tempo emit a separate metric for 429 responses.
      - alert: ObjectStoreThrottling
        expr: |
          sum by (op) (
            rate(loki_objectstore_request_duration_seconds_count{
              status_code="429"
            }[5m])
          ) > 0
        for: 10m
        labels:
          severity: warning
          team: platform

      # 7. Loki ingester stream count approaching the limit.
      #    Catches the case where Loki is dropping logs because
      #    the per-tenant stream limit is hit.
      - alert: LokiStreamsHigh
        expr: |
          sum by (tenant) (loki_ingester_streams)
            > 50000
        for: 10m
        labels:
          severity: warning
          team: platform

      # 8. Boltdb-shipper upload backlog.
      #    A persistent backlog means the ingester cannot flush
      #    to the object store fast enough.
      - alert: LokiBoltdbShipperBacklog
        expr: |
          rate(loki_boltdb_shipper_uploader{uploaded="false"}[10m]) > 0
        for: 15m
        labels:
          severity: warning
          team: platform

Retention enforcement

The cheapest prevention for disk fill is a correct retention configuration. For Prometheus:

# /etc/default/prometheus -- retention is set by flag. There
# is no retention key in prometheus.yml.
# 30 days retention - tune to your retention SLO.
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"
# The size cap deletes the oldest blocks. It does not stop
# writes.

The --storage.tsdb.retention.size cap is the safety belt. Without it, Prometheus will write until the disk is full and crash on write. With it, Prometheus drops the oldest blocks once the size limit is reached.

For Loki, retention is enforced by the compactor:

# /etc/loki/config.yaml
compactor:
  working_directory: /data/loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

limits_config:
  retention_period: 720h
  # 30 days

How to validate it

Confirm the TSDB is on a separate filesystem with adequate inodes:

# READ-ONLY
df -h /var/lib/prometheus
df -i /var/lib/prometheus

A healthy output shows at least 30 percent free space and at least 100,000 free inodes. The exact thresholds depend on the provisioned capacity.

Confirm the storage size metrics match the filesystem:

# READ-ONLY
curl -s 'http://prom-primary.internal:9090/api/v1/query?query=prometheus_tsdb_storage_blocks_bytes' \
  | jq '.data.result[0].value[1]'
du -sh /var/lib/prometheus

The two numbers should be within 10 percent of each other (du counts directory metadata; the metric counts only block files). A larger gap means the TSDB has accumulated non-block files that the metric does not account for - often query-tracker files or stale lock files.

Confirm the object store is reachable from the Loki host:

# READ-ONLY
curl -s http://loki.internal:3100/ready
# Returns "ready" if the ingester is up and the storage
# backend is reachable.

Confirm retention is actually deleting old blocks:

# READ-ONLY
ls -lh /var/lib/prometheus/data
# Look at the block directories; the oldest should be
# roughly retention.time old.

Inject a retention test:

# CONFIGURATION - reduce retention to 1 hour, restart, watch.
# Retention is a startup flag, so a reload does not pick the
# change up; the process has to be restarted.
sudo sed -i 's/retention.time=30d/retention.time=1h/' /etc/default/prometheus
sudo systemctl restart prometheus
# Wait, then:
ls /var/lib/prometheus/data | wc -l
# Restore.
sudo sed -i 's/retention.time=1h/retention.time=30d/' /etc/default/prometheus
sudo systemctl restart prometheus

A working setup shows the block count drop to a smaller value within the hour, then climb back to a steady state after the restore.

How it can fail

1. Disk fills, Prometheus crashes

node_filesystem_avail_bytes crosses zero; Prometheus emits a write error; the process terminates. Symptom: the process is down; the WAL does not advance; alerts that were firing during the outage are lost. Action: free disk (delete old blocks manually if retention.time was insufficient), restart Prometheus, allow the WAL to replay.

2. WAL inode exhaustion

The filesystem runs out of inodes before it runs out of bytes. Symptom: node_filesystem_files_free near zero; Prometheus log shows “no space left on device” on a filesystem with bytes free. Action: recreate the filesystem with a smaller -i ratio, or move the WAL to a dedicated filesystem with adequate inodes.

3. Object store 403 (credentials expired)

The IAM role or service account used by Loki/Tempo to write to the bucket has been revoked. Symptom: loki_objectstore_request_duration_seconds_count \{status_code="403"\} rises; ingester flushes fail; loki_boltdb_shipper_uploader backlog grows. Action: rotate the IAM credentials; update the bucket policy; confirm a manual write to the bucket succeeds.

4. Object store throttling

The bucket is being read or written at a rate that exceeds the per-prefix limits (S3 returns 503 SlowDown). Symptom: loki_objectstore_request_duration_seconds_count \{status_code="429"\} rises; query latency climbs. Action: add a request-rate budget per prefix; use a dedicated prefix for the platform’s bucket; consider sharding across more prefixes.

5. Loki ingester stream limit

A single tenant exceeds max_streams_per_user (default 100,000). Symptom: loki_ingester_streams{tenant} at the ceiling; new streams are dropped; the dropped-streams counter increments. Action: identify the offending labels, add a stream selector to the pipeline to drop high-cardinality labels before ingestion.

6. Retention not enforced

The retention setting is configured but the compactor (Loki) or TSDB (Prometheus) is not enforcing it - either because the compactor is down or because the retention_enabled flag is missing. Symptom: the bucket keeps growing; the cloud bill climbs. Action: confirm the compactor is running, confirm the flag is set, confirm retention_delete_delay has elapsed.

7. Bucket policy denies writes

A bucket policy was applied that allows reads from the ingester VPC but denies writes. Symptom: Loki ingester flushes fail with 403; the bucket size stops growing but the ingester’s in-memory queues grow. Action: update the bucket policy; confirm both read and write are allowed from the ingester’s IP range.

How to troubleshoot it

Security implications

The Prometheus TSDB contains every metric the platform has ever collected. It is a privileged data surface: labels often include user IDs, account IDs, request paths, and other PII. The disk that holds the TSDB should be encrypted at rest; the TSDB directory itself should be readable only by the Prometheus process user.

The object store bucket is the same data shape, scaled out. The bucket policy should allow only the platform’s IAM role to write and only the query path’s IAM role to read. Public buckets are the canonical breach shape for object stores.

Retention is a security control as well as a cost control. A platform that retains logs indefinitely retains the breach forever. The retention SLO and the legal-retention requirement are usually different; document the longer of the two and enforce it.

Performance implications

Disk is the dominant cost for a Prometheus deployment. The right sizing pattern is:

• TSDB disk. Provision for the steady-state prometheus_tsdb_storage_blocks_bytes plus the WAL plus one full compaction’s worth of headroom. A 30-day retention at 10,000 active series is roughly 50-200 GiB depending on label cardinality.
• Object store. Provision by query and ingest rate, not by data volume. The bucket cost is dominated by request count and storage class; the data volume is a rounding error.
• Loki ingester memory. Provision for max_streams_per _user times the per-stream overhead. A 100,000-stream ingester needs roughly 4-8 GiB of resident memory.

The retention SLO drives disk sizing. A 90-day retention is roughly 3x the cost of a 30-day retention. The trade-off is “how far back does the investigation need to go?”.

Production guidance

• Set a retention.size cap on every Prometheus deployment. The cap is the safety belt; without it the disk fills and Prometheus crashes.
• Provision TSDB disk for 2x the expected steady-state blocks size. The headroom absorbs a cardinality spike without paging.
• Alert on predict_linear(node_filesystem_avail_bytes, ...) for a 7-day projection, not on absolute percent. A 50 percent disk that fills in a week is a page; a 90 percent disk that fills in a year is not.
• Configure retention_enabled: true and compactor.retention_delete_delay on every Loki deployment. The retention flag without the compactor is a no-op.
• Encrypt the TSDB disk at rest. The metric labels often contain PII.
• Restrict the object store bucket to platform IAM roles only. The bucket should not be world-readable.

Verification

You should now be able to answer:

• What are the four storage surfaces in a Prometheus stack?
• Why does predict_linear over disk-usage series give a better alert than an absolute percentage threshold?
• What is the difference between TSDB disk exhaustion and WAL inode exhaustion?
• What does the status_code label on an object store request tell you?
• Why is retention.size a safety belt and not a retention policy?

Quiz