Unlimited Retention

The storage bill arrives on the first of the month. Last quarter it was $4,200. This quarter it is $11,800. The CFO asks what changed. Nothing changed. The cardinality is the same. The number of services is the same. The retention policy, set to the default, has been quietly accumulating data for fourteen months. The disk fill rate is linear; the bill is linear; the budget for observability was set on the assumption of constant cost. The conversation with the CFO ends with the directive to cut the retention. Nobody has tested what cuts at which tier. The team picks a number. Six months. A quarter of the long-tail incidents from before that date become un-investigatable.

This is the unlimited retention anti-pattern. The pattern is not the existence of stored data; the pattern is the absence of a decision about how long data lives, where it lives, and what it costs at each tier. The cost curve is not linear in time; it is linear in data volume, which grows linearly with retention. The decision deferred at deploy time is forced at finance time.

What it is

The unlimited retention anti-pattern is the practice of storing telemetry with no explicit retention policy, or with a retention policy set to the maximum the storage backend permits. The default Prometheus retention is fifteen days; the default Loki retention is the same; the default Tempo retention is set by the object storage lifecycle. None of the defaults are wrong; all of them are decisions deferred. A team that has not documented a retention policy has implicitly accepted the default, and the default is rarely the right answer for the workload.

Compare to the alternative: tiered retention. The four tiers are well-defined in the literature.

Hot        Last 24h         SSD or NVMe         Full resolution
                                                     |
                                                     v
Warm       Days 2 to 30     Local disk or        Downsampled or
                            object storage        full resolution
                                                     |
                                                     v
Cold       Days 31 to 365   Object storage       Downsampled, 5m
                            (S3, GCS, Azure)      and 1h blocks
                                                     |
                                                     v
Frozen     Older than 365   Object storage,      Downsampled,
                            infrequently read    compressed

The tier is matched to the question. What was the error rate at 14:32 yesterday? is a hot-tier question. What was the error rate in week 22? is a warm-tier question. Did the system behave correctly in 2024? is a cold-tier question. What did the platform look like in 2020? is a frozen-tier question, and the answer is rarely needed.

The trade-off is honest. Tiered retention costs you the long-tail historical investigation: the failure mode that appears once a year and requires last year’s data to characterise. The mitigated cost is the unbounded monthly bill and the operational complexity of running a storage backend that holds more than it should.

Why a sysadmin cares

The cost of unbounded retention is paid in five places, all of which are visible to the operator on call.

Storage cost. The headline number. Object storage is cheap per gigabyte, but the volume at fifteen-month retention for a moderate Prometheus deployment (twenty-six million active series, two-week retention) is on the order of one terabyte; at fifteen months, it is thirty terabytes. The monthly bill rises proportionally.

Query cost. The query cost grows with the data scanned. A range query over a year of full-resolution metrics is prohibitively expensive even on warm storage. The query times out; the operator works around the timeout by re-querying with a narrower range; the investigation takes longer.

Restore cost. The cost of restoring a frozen-tier query is minutes to hours, depending on the storage backend. The investigation that needs frozen data is paused while the data is thawed.

Compliance cost. Some data must not be retained beyond a regulatory window. Logs containing PII, audit logs, and EU customer data have explicit retention limits. A platform that retains everything retains the data that should have been deleted. The compliance exposure grows linearly with retention.

Cognitive cost. The team that does not know what their retention is does not know what their cost is, does not know what their query budget is, does not know what their compliance exposure is. The decisions are deferred until a finance or audit conversation forces them.

How it works

The retention decision is a function of three questions. The answers determine the tier and the storage backend.

Question 1: How old is the oldest incident you have investigated
            in the past year?
                |
                v
Question 2: How old is the oldest compliance evidence you have
            been asked to produce?
                |
                v
Question 3: How old is the oldest capacity-planning model you
            have built from telemetry?
                |
                v
            Tier | Window  | Backend  | Resolution
            -----|---------|----------|------------
            Hot  |  24h    | SSD      | Full
            Warm |  30d    | Disk     | Full or 5m
            Cold |  365d   | Object   | 5m, 1h
            Frozen| 7y     | Object   | 1h, archived

The four answers are usually different. The longest answer wins. The longest answer is usually Question 2 (compliance evidence for a financial-services workload is seven years). The shortest answer is usually Question 3 (capacity planning rarely needs more than ninety days). The platform is sized to the longest answer; the cost is not linear in the longest answer, because the bulk of the data lives in the cold and frozen tiers where resolution is downsampled.

How to configure it

The configuration is per backend. The hot-tier and warm-tier configuration is on the storage host; the cold-tier configuration is on the compactor; the frozen-tier configuration is on the object storage lifecycle.

# Prometheus 2.55.x: hot + warm tiers only.
# /etc/default/prometheus -- hot-tier retention is a flag.
# There is no retention key in prometheus.yml.
# 15 days is the default; 30 days is the upper bound for a
# single-host Prometheus with a single TSDB. Beyond this, the
# WAL replay time on restart is measured in minutes.
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d"

# /etc/prometheus/prometheus.yml
global:
  external_labels:
    cluster: prod-eu

# Long retention is achieved via Thanos or Mimir, not via
# Prometheus. The Thanos-sidecar uploads the WAL to S3;
# the compactor downsamples; the store serves queries.

# thanos-store.yml: cold tier
type: S3
config:
  bucket: thanos-cold
  endpoint: s3.eu-west-1.amazonaws.com
  access_key: '<aws-access-key>'
  secret_key: '<aws-secret-key>'

# Loki 3.x: cold tier via the compactor.
# /etc/loki/config.yaml
limits_config:
  retention_period: 744h       # 31 days for the default tenant
  retention_stream:
    - selector: '{namespace="prod"}'
      period: 2160h             # 90 days for production
    - selector: '{namespace="dev"}'
      period: 168h              # 7 days for development

compactor:
  retention_enabled: true
  delete_request_store: filesystem
  working_directory: /loki/compactor

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13

# Tempo: cold tier via the compactor.
# /etc/tempo/config.yaml
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.eu-west-1.amazonaws.com
      access_key: '<aws-access-key>'
      secret_key: '<aws-secret-key>'

compactor:
  compaction:
    block_retention: 168h        # 7 days for traces
  ring:
    kvstore:
      store: memberlist

The three configurations share a discipline: the retention is expressed in hours, the selector matches the tenant or the namespace, the cold tier lives in object storage, and the cost is documented.

How to validate it

Four commands confirm the retention policy is in force.

# 1. Prometheus oldest block on disk.
# Severity: READ-ONLY
ls -la /prometheus/data/chunks_head/ | head -3

The block modification timestamp older than the retention period indicates a misconfiguration.

# 2. Loki retention enforcement.
# Severity: READ-ONLY
curl -s http://loki-distributor:3100/config \
  | jq '.limits_config'

The retention_period should match the documented policy.

# 3. Thanos store: oldest block.
# Severity: READ-ONLY
thanos store info \
  --objstore.config-file=/etc/thanos/store.yml \
  | head -5

# 4. Storage cost attribution. The CFO-friendly view.
# Severity: READ-ONLY
aws s3api list-objects-v2 \
  --bucket thanos-cold \
  --prefix '01H' \
  --query 'sum(Contents[].Size)' \
  --output text

The byte count, multiplied by the per-gigabyte cost, is the monthly bill. The bill should match the budget; if not, the retention policy is the lever.

How it can fail

Five shapes recur when the retention decision is not made explicitly.

1. The default retention drift. The Prometheus retention flag is unset; the default is fifteen days. The team assumes thirty. The storage cost is double the assumption. The quarterly review finds the discrepancy.
2. The single-host TSDB exhaustion. A single Prometheus instance holds thirty days of high-cardinality metrics. The WAL replay time on restart is twenty minutes. The platform is unavailable during deploys. The fix is to move to a Thanos or Mimir-sidecar architecture and tier the storage.
3. The Loki stream that never flushes. A bad-label incident (Lesson 03 of the Loki Labels chapter) produces millions of streams. The chunks never fill. The ingester holds them in memory. The compactor never deletes them. The storage cost grows unboundedly until the label is fixed.
4. The compliance retention mismatch. Logs are retained for one year to satisfy an audit requirement; metrics are retained for the same period for consistency. The compliance requirement was for the audit logs only. The metrics retention is paying for storage that the requirement does not demand.
5. The frozen-tier thaw storm. A long-tail incident requires data from eighteen months ago. The frozen tier is queried by thirty engineers simultaneously. The thaw cost is a four-hour query budget. The investigation pauses.

How to troubleshoot it

1. Quantify the storage: bytes per tier, cost per tier,
   retention per tier
        |
        v
2. Quantify the demand: queries older than 30d, queries
   older than 90d, queries older than 1y
        |
        v
3. Match the tiers to the demand
        |
        +-- hot: keep all queries within 24h
        |
        +-- warm: keep queries 1d-30d
        |
        +-- cold: keep queries 30d-365d, downsampled
        |
        +-- frozen: keep queries 1y+, archive
        |
        v
4. Implement in object-storage lifecycle rules and
   backend compactor config
        |
        v
5. Validate with cost report and storage growth rate

Security implications

Retention is a security boundary. Logs that contain PII cannot be retained beyond the window the regulator permits. Metrics that contain user identifiers cannot be retained beyond the same window. Trace data that contains request bodies must be scrubbed before retention. The retention policy is the enforcement mechanism for the data-handling policy. A misconfigured retention is a compliance incident waiting to be reported by an auditor, not by the team.

Performance implications

The query performance of a Prometheus server is set by the size of the head block and the size of the most recent persisted blocks. The performance of a Loki cluster is set by the index fan-out and the chunk cache. Long retention with full resolution degrades both. Downsampling at the cold tier is the right trade-off: the storage cost falls by an order of magnitude; the query resolution coarsens; the investigations that need full-resolution data at long range are rare.

Verification

You should now be able to answer:

• What are the four retention tiers, and what question does each tier answer?
• Why is the cost curve of unbounded retention non-linear in time but linear in volume?
• Where in the stack is the cold-tier retention enforced for Loki, Tempo, and Prometheus respectively?
• What is the relationship between retention policy and compliance exposure?

Quiz