Why Long-Term Metrics Storage

A capacity-planning meeting in October. Engineering asks: “what was CPU pressure on the database hosts last November, the week before Black Friday?” Nobody has that data. The default Prometheus retention is 15 days, the cluster was rebuilt in April, and the question now drives a six-week backfill project or a 12-month wait. The data was always being scraped; it just stopped existing.

This is the operational shape of long-term metrics storage. The problem is never “we cannot collect the data”; it is “we did not plan for the data we will be asked about”. This lesson frames the trade-off, the cost curve, and the failure modes so the decision is made before the question arrives.

What it is

Long-term metrics storage is the discipline of keeping Prometheus samples beyond the default 15-day horizon. It is a trade-off: every additional month of retention costs disk, memory, query latency, and operational complexity, and the cost is paid whether or not anyone ever looks at the data. The right answer depends on which operational questions require old data and how often they are asked.

Three storage tiers appear in a Prometheus deployment. They are not equivalent:

• Local TSDB — embedded in every Prometheus server, scoped to one host, single-writer. Holds recent data for fast queries.
• Remote store — a long-term destination reached by remote_write (Thanos, Mimir, Cortex, InfluxDB, vendor services). Holds data for weeks to years; queries are federation-style aggregations.
• Cold archive — object storage with infrequent queries, used for compliance and forensic searches (Thanos shipper to S3 with long retention; or a separate frozen tier in Mimir).

The tiers overlap in purpose. The mistake is to treat them as interchangeable, or to assume one tier scales to all three workloads.

Why a sysadmin cares

Four operational questions drive the decision. None of them are “monitoring” questions; they are investigation, planning, and forensics questions that arrive at unpredictable cadences.

1. Year-over-year and seasonality. Was last quarter worse than the same quarter last year? A retailer asking this in November needs data from November last year. A SaaS business asking it in March needs data from March last year. Without 12 months of history, the answer is a guess.
2. Capacity planning at human time scales. Disk, network, and licence budgets are planned weeks and quarters ahead. The data you need to plan them is the data from last season.
3. Anomaly detection baselines. Most statistical and ML-based detectors need at least one full seasonality cycle — often a week, often a year — of history to set thresholds without a cold-start period.
4. Forensic search. “Did the change at 02:00 last month correlate with the slow drift the user is complaining about today?” Without retention, the question is unanswerable. With retention, the answer takes an hour.

The fifth reason, rarely admitted, is compliance. Regulated workloads (PCI-DSS, financial services, healthcare audit trails) frequently require metrics retention that has nothing to do with operations and everything to do with auditors.

How it works

The Prometheus data path is two-tiered by default:

            Scrape (15 s)
                 |
                 v
            Local TSDB  ----->  /api/v1/query  (recent, fast)
            (15 days,   ----->  recording rules
             single host)
                 |
                 | remote_write
                 v
            Remote store ----->  global query    (weeks to years)
            (Thanos / Mimir
             / Cortex)
                 |
                 v
            Cold object store
            (S3 / GCS / Azure Blob,
             years, infrequent)

The local TSDB holds recent samples in head + persisted blocks. The two retention knobs are --storage.tsdb.retention.time (default 15d) and --storage.tsdb.retention.size (default disabled). The remote store holds copies of the same samples written over the network.

The two tiers are not equivalent. The local TSDB is the primary; remote_write is a write-ahead replication. If remote_write fails, the local TSDB still has the data until retention expires. If the local TSDB dies and the WAL was not replayed, the data is gone from remote too — unless an HA replica was writing.

The cost curve

The numbers that decide whether a single host can hold your retention are surprisingly predictable. The dominant variable is active series: distinct time series that are being appended to right now.

bytes per sample         ~ 1-3 bytes  (varies with label cardinality)
samples per series / day  ~ 17 280     (15 s scrape interval)
bytes per series / day    ~ 17-52 KB

So a fleet emitting 5 million active series at 15 s scrape writes roughly 85-260 GB per day, before considering the WAL, the in-memory head, or compaction overhead. At 30 days that is 2.5-7.8 TB of compacted data on disk; the head block, page cache, and rule evaluation need a further 32-64 GB of RAM on the same host. The disk fills first; the memory follows.

The same fleet at 5 s scrape writes three times as much. The same fleet with 50 million active series writes ten times as much. The curve is brutal because every doubling of either axis doubles storage cost, and every tenfold multiplies it.

How to configure it

Two flags, deliberately set:

prometheus \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=1.5TB \
  --storage.tsdb.wal-compression

Both retention knobs should be set on any disk-constrained host: time encodes policy (“we will not keep less than X days”), size is the circuit-breaker (“we will not keep more than Y bytes”). Setting only time means a misjudged scrape fan-out fills the disk; setting only size means an admin who forgets to enforce retention keeps data forever. Set both.

Beyond the flags, the decision tree:

Active series    Local disk headroom    Decision
-------------    --------------------    ---------------------------------
less than 2 M    greater than 5 TB       local 30-90 days is fine
2-10 M           2-10 TB                 local + remote_write to object store
10 M plus        any                     remote_write is mandatory; local
                                        holds 6-24 hours of recent only

The thresholds are not universal. A fleet with low-cardinality metrics (kubelet, node_exporter, standard exporters) at 15 s scrape can do 5 M active series on a beefy host. A fleet with high-cardinality business metrics (per-user, per-tenant, per- request) hits the wall much earlier.

How to validate it

# Active series right now (the dominant cost variable)
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.headStats.numSeries'

# Blocks on disk and their time spread
ls /var/lib/prometheus | grep '^01' | wc -l
du -sh /var/lib/prometheus/

# The retention horizon it is actually enforcing
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.headStats.minTime'   # not older than retention.time

Three queries that prove the engine is keeping what you asked for:

# Smallest maxTime across persisted blocks: that is your retention horizon
min(prometheus_tsdb_head_min_time)

# Active series count, scraped
prometheus_tsdb_head_series

# Oldest sample still queryable, in days
(time() - min(prometheus_tsdb_head_min_time)) / 86400

How it can fail

1. Disk full, no size retention. Symptom: compactions_failed_ total climbs, “no space left on device” in the logs, ingestion stalls, and the head cannot truncate. A single Prometheus cannot keep its data; the symptom is also the cause.
2. Retention too short to answer the question. Symptom: the data was always there, then it was not. A capacity-planning question becomes a “we did not keep that long” answer. The failure is invisible at the time; it appears in a post-mortem six months later.
3. Size retention set below growth curve. Symptom: historical queries silently return only a few days; the engine has dutifully deleted the blocks you wanted. It did exactly what you asked.
4. Scrape fan-out grown faster than capacity. Symptom: the retention horizon shrinks day by day without anyone changing the flag. The cause is upstream: a new exporter, a new service, a label added to existing metrics. The fix is upstream (relabel, drop, or cardinality budget), not bigger disk.
5. WAL replay after a long outage. Symptom: Prometheus was down for an hour; on restart, it spends 30 minutes replaying WAL. If the WAL filled the disk during the outage, replay fails and recent data is lost. A larger retention time without WAL disk space planning makes this worse, not better.
6. HA replica silence. Two Prometheus writing the same remote_write target, one silent for days. Symptom: the remote store is missing every-other sample for that replica’s labels; deduplication at query time hides the gaps as “noise”. The failure mode is invisible until a forensic question needs the missing data.

How to troubleshoot it

1. Is the engine enforcing retention as configured? min(prometheus_tsdb_head_min_time) tells you the oldest data it is willing to serve. Compare to now() - retention.time.
2. Is the disk the constraint or the cardinality? du -sh /var/lib/prometheus/ versus prometheus_tsdb_head_ series versus the bytes-per-series table above. Big disk + few series = underused hardware. Small disk + many series = the cost curve has caught up with you.
3. What is the cardinality leader? /api/v1/status/tsdb returns seriesCountByMetricName. The top metric by series count is where the budget is going.
4. Where is the remote store, if any? curl remote-store/ api/v1/labels should list the same label set the local Prometheus is shipping. Mismatches are the first sign of a misconfigured remote_write.
5. Logs. Grep for “compaction”, “overlapping blocks”, “retention”. The engine narrates its own retention decisions.

Security implications

• The TSDB contains every label. Hostnames, usernames, tenant IDs, request IDs, occasionally worse. Long retention widens the exposure window. Treat the data directory as you would treat a database backup: 0700/0750, encrypted offsite, audit access.
• Query access is read access to all retained data. A Grafana user with metrics:read against a long-retention store can query labels that were present months ago and have since been removed from the codebase. This is sometimes the point; it is sometimes the leak. Audit query access alongside retention.
• Compliance and right-to-erasure. Long retention conflicts with privacy regimes that require deletion on request. A label value identifying a user is still identifying that user six months later. The recording-rule-and-drop pattern (record a derivative, then drop the high-cardinality original after N days) is a compromise; the strict answer is to never have high-cardinality personal data in metric labels.

Performance implications

Disk usage is roughly 1-3 bytes per sample on average, dominated by series count and scrape interval. Memory tracks active head series, not total stored data. Churn (short-lived series that appear and disappear) is the expensive shape because each one forces the index to keep the label set live. Compaction is an I/O burst every two hours; on starved disks it coincides with query latency spikes, which is normal, not a fault.

Verification

You should now be able to answer:

• What does the 15-day default actually buy you, and what does it not?
• At what cardinality does a single Prometheus stop being a viable long-term store?
• What does “no downsampling” mean for the cost of a 90-day retention?
• What is the difference between a remote store and a cold archive, and which questions does each answer?

Quiz