Metrics Capacity

A monitoring host has 64 GiB of RAM. The platform team declared it “enough for a year of growth.” Six months later Prometheus was OOM-killed twice in a single week. The post-mortem found 11 million active series, a planning figure of 4 KiB per series, and a host that should have held 12 million series cleanly. It held 7 million cleanly and another 4 million in swapping death. The arithmetic was right; the planning figure was wrong for this workload.

This lesson is the arithmetic. It is the same arithmetic in every capacity review, and it is the arithmetic most often done from memory instead of from metrics.

What metrics capacity is

Metrics capacity is the answer to four questions about a Prometheus host, derived from the workload rather than from the vendor data sheet:

1. How many active series can the head block hold without crossing the OOM line?
2. How many samples per second can the ingestion path accept without dropping scrapes?
3. How many bytes on disk will the on-disk blocks occupy over the retention window?
4. How much headroom is left after the steady-state workload is subtracted from the host budget?

The four numbers are linked. Series drives memory. Sample rate drives ingest CPU and disk. Retention drives disk and query cost. Headroom is the difference between the budget and the steady state on each axis.

The common mistake is to answer only one of the four and plan capacity against it. A host that is sized for memory frequently cannot keep up with the scrape rate; a host sized for scrape rate frequently runs out of disk before retention expires.

Why a sysadmin cares

The cost of misjudging metrics capacity falls on three operational lines:

• Availability. Prometheus OOMs are noisy. Each kill triggers a WAL replay, a gap in rule evaluation, and a customer-visible dashboard outage. The pattern repeats until the budget is corrected.
• Query latency. Rule evaluation, head garbage collection, and compaction all scale with series count. A Prometheus that was snappy at one million series is visibly ill at five million. The slowdown is not flagged as an incident until an alert fires late.
• Remote-write bills. If the platform remote-writes to a managed backend, the per-series-per-second rate is the line item. Doubling series without doubling the budget doubles the invoice.

The arithmetic is small. The discipline of running it from live metrics, not from a previous engineer’s spreadsheet, is what separates a capacity review that prevents outages from one that records them.

How it works: the four equations

Each axis has one equation. The four together describe the platform.

  active_series           = sum of unique (metric, labels) in the head
  memory_budget_bytes     = active_series  *  bytes_per_series
  samples_per_second      = active_series  /  scrape_interval_seconds
  disk_budget_bytes       = samples_per_second  *  retention_seconds
                             *  bytes_per_sample_on_disk
  headroom_fraction       = (host_budget  -  steady_state)  /  host_budget

bytes_per_sample_on_disk is the figure after chunk compression: Prometheus stores roughly one to two bytes per sample once a chunk is written. Compression is already inside that constant, so it is never divided out a second time. A disk budget that applies a compression ratio on top of a one-to-two-byte figure is wrong by that ratio, and it is wrong in the direction that runs the volume out of space.

For a workload of 5 million active series, scraped every 15 seconds, retained 30 days, with the planning figures of 4 KiB per series and 2 bytes per sample on disk, on a 64 GiB host reserved 40 GiB to Prometheus:

  memory_budget  =  5 000 000  *  4 KiB              =  ~20 GiB
  samples/sec    =  5 000 000  /  15 s               =  ~333 000 samples/s
  retention      =  30 d  *  86 400 s                =  2 592 000 s
  disk_budget    =  333 000  *  2 592 000  *  2 B
                 =  ~1.73 TB  (~1.57 TiB) on disk
  headroom       =  (40 GiB  -  20 GiB)  /  40 GiB   =  50%

At the optimistic end of the range — one byte per sample — the same workload needs ~0.86 TB. The range is the difference between a 1 TB volume and a 2 TB volume, which is why the constant is measured rather than assumed.

Memory is the term that kills the process; disk is the term that is easiest to under-provision. The same workload at 10 million active series needs 40 GiB for the head alone, before WAL, queries and OS overhead — a host of 96 GiB reserved to Prometheus is the defensible minimum — and it doubles the disk budget to roughly 3.5 TB.

How to configure it

A metrics capacity plan is four numbers and a tripwire per number.

1. The host budget. What you reserve to Prometheus out of the host’s physical RAM:

# /etc/default/prometheus
ARGS="--storage.tsdb.path=/var/lib/prometheus/data \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=0 \
      --storage.tsdb.wal-compression \
      --query.max-concurrency=20"

The memory limit is enforced by the container manager (Docker, systemd with MemoryMax=, Kubernetes resources.limits.memory), not by Prometheus itself. Set the container limit to 75% of the host’s reserved memory to leave room for the OS and the exporters.

2. The per-job cap. What each scrape job is allowed to contribute. Use sample_limit and label limits to fail the scrape at the source when a job misbehaves:

# prometheus.yml
scrape_configs:
  - job_name: checkout-api
    scrape_interval: 15s
    static_configs:
      - targets: ['10.0.1.4:8080', '10.0.1.5:8080']
    sample_limit: 20000
    label_limit: 40
    label_name_length_limit: 120
    label_value_length_limit: 512
    # Keep the worst contributor bounded. A breached sample_limit
    # fails the scrape and increments
    # prometheus_target_scrapes_exceeded_sample_limit_total —
    # loud at the source, silent in the head.

3. The platform cap. What the Prometheus instance commits to holding, expressed as a series ceiling:

# rules/capacity.yaml
groups:
  - name: capacity-budget
    rules:
      - record: job:active_series:count
        expr: count by (job) ({__name__=~".+"})

      - alert: HeadSeriesApproachingCap
        expr: prometheus_tsdb_head_series > 5_600_000
        for: 15m
        labels: {severity: critical, team: observability}
        annotations:
          summary: 'Head series above 80 percent of the 7M platform cap'
          description: |
            Active series is {{ $value | humanize }}. The headroom
            target is 7M - (5M steady) = 2M. Crossed 80 percent of
            the target.

      - alert: JobCardinalityOverBudget
        expr: job:active_series:count > 50_000
        for: 30m
        labels: {severity: warning, team: observability}
        annotations:
          summary: 'Job {{ $labels.job }} above its 50k series budget'

4. The retention pair. The on-disk retention in time and the disk-volume cap. The first bounds age; the second bounds volume:

; /etc/default/prometheus
ARGS="... --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=2TB ..."

The retention.size cap is the backstop for a series growth that outran the time-based cap. Set it to the disk volume minus the 30% headroom the next lesson defines. The figure above is sized for the worked example: 30 days of a 5-million-series workload is roughly 1.73 TB, so a 2 TB cap on a 3 TiB volume holds the retention window and still leaves the headroom. The size flag takes base-2 units, so 2TB is 2 TiB.

How to validate it

Confirm the four numbers from the live platform.

# READ-ONLY: head series — the live cardinality gauge.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_head_series' | jq .

# Worst metric names, by series count.
topk(10, count by (__name__) ({__name__=~".+"}))

# Worst jobs, by series count.
topk(10, count by (job) ({__name__=~".+"}))

# Scrape rate per job, summed.
sum by (job) (rate(scrape_samples_scraped[5m]))

# READ-ONLY: per-block on-disk analysis.
promtool tsdb list /var/lib/prometheus/data
promtool tsdb analyze /var/lib/prometheus/data 01J8K3XW7M0R2Y3T4V5B6N8Q9
# Reports per-block series count and the highest-cardinality
# label pair. Useful for an offline investigation.

The planning figure for bytes per series:

# READ-ONLY: derived per-series resident memory.
# Take the ratio during steady state — not during a spike.
echo "scale=2; $(curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=process_resident_memory_bytes' \
  | jq '.data.result[0].value[1]') \
  / $(curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_head_series' \
  | jq '.data.result[0].value[1]')" | bc
# Expected: somewhere in 3000-8000 bytes per series for
# a workload with moderate label-set size.

Write the result into the capacity file next to the budget:

# capacity.yaml — owned by the observability team
hosts:
  prometheus-1.internal:
    memory_limit_gib: 40
    planning_bytes_per_series: 4096   # re-measure quarterly
    platform_cap_active_series: 7_000_000
    warn_fraction: 0.8
    retention_time: 30d
    retention_size: 2TB               # 30d of 5M series is ~1.73 TB
    disk_volume: 3TiB

How it can fail

1. Slow series creep. Active series grows 5% a week for months. No alert fires because none is set; the first symptom is a swap storm on a Tuesday afternoon. Symptom: prometheus_tsdb_head_series on a 90-day graph is a staircase, not a step.
2. The instant explosion. A deploy introduces a metric labelled by customer identifier. Series count doubles in an hour; OOM within the same coffee. Symptom: near-vertical prometheus_tsdb_head_series graph starting at the deploy marker.
3. The scrape-rate bottleneck. The host is sized for memory but the scrape pool exceeds what ingestion can accept. Scrape duration crosses the interval; targets drop out. Symptom: scrape_duration_seconds > scrape_interval together with up == 0 flapping on targets that were up before.
4. Disk fill before retention expires. A series surge pushes on-disk blocks past the volume size cap; Prometheus starts deleting the oldest blocks to enforce retention.size regardless of age. Symptom: prometheus_tsdb_storage_blocks_bytes at 100% of the disk-volume cap; older-than-expected blocks disappear on the timeline.
5. Compaction storm. After a series surge the head must compact. Compaction competes with ingestion for CPU and I/O; scrapes start timing out. Symptom: prometheus_tsdb_compactions_total spiking together with scrape_duration_seconds crossing the interval.
6. WAL replay purgatory. After an OOM kill, WAL replay of a large head takes tens of minutes; the boot is killed by a watchdog and retried. Symptom: Prometheus logs replay WAL progress lines for tens of minutes; prometheus_target_sync_failed_total rising through the replay.

How to troubleshoot it

The order matters. Cheapest diagnostic first.

1. Which axis is the problem? Read prometheus_tsdb_head_series, process_resident_memory_bytes, and prometheus_tsdb_storage_blocks_bytes together. The axis closest to its cap names the failure.
2. Find the contributor. If cardinality is the cap, topk(10, count by (job) ({__name__=~".+"})) shows the worst job. Drill in with count by (job, instance) (worst_metric_name) and then count by (suspect_label) (worst_metric_name).
3. Correlate with change. Overlay deploy annotations in Grafana; check process_start_time_seconds for recently restarted targets; ask what shipped.
4. Decide: contain or evict. If the platform is stable, fix forward with relabel rules. If the host is thrashing, mitigate first; lesson 6 in this module is the runbook.
5. Confirm the next restart will be safe. Check WAL size: du -sh /var/lib/prometheus/data/wal. A WAL above ~50% of memory means the next restart will be slow or will OOM during replay.

Security implications

Cardinality is an availability boundary, and availability is a security property. Any endpoint that can influence label values — a user-controlled URL path echoed into a label, an unauthenticated /metrics that accepts query parameters — is a denial-of-wallet or denial-of-service vector against the monitoring platform. The sample_limit and label limits are therefore also security controls, not just capacity controls.

Label values are also data. A label that carries user identifiers replicates those identifiers into the TSDB, into remote-write receivers, into backups, and into snapshot exports. Treat the metrics pipeline as a data store in its own right when classifying what may flow through it; the platform security part of the course covers authentication and transport for each endpoint.

Performance implications

• Memory. Resident cost is dominated by the series table, postings, and open head chunks. Plan against 3-8 KiB per active series as a starting range; measure yours.
• CPU. Ingestion cost per sample is modest (prometheus_tsdb_head_samples_appended_total tells you the rate). The cardinality-driven CPU costs are head GC, compaction, and rule evaluation over many series.
• Disk. WAL and blocks grow with churn; du the data directory weekly and graph it.
• Query cost. Queries that touch many series (sum by () over millions of inputs) get slower roughly linearly; the capacity budget indirectly protects dashboard latency.

The trade-off of a tight budget: teams sometimes aggregate away a dimension they wanted (per-endpoint latency becomes per-route-group latency). That is a real observability loss. The budget forces the loss to be a conscious one priced in series, rather than an unconscious one priced in RAM.

Production guidance

• Derive the bytes-per-series constant from your own metrics, not from a blog post. Re-derive it quarterly.
• Set the platform cap from memory, not from hope. With a planning figure of 4-8 KiB per series and a 64 GiB host reserved 40 GiB to Prometheus, the defensible cap is roughly 5-7 million series. Confirm against the live ratio and revise.
• Set sample_limit on every scrape job. A misbehaving job failing its scrape is better than a misbehaving job taking the platform with it.
• Alert at 80% of the platform cap and 80% of the per-job budget. The 80% line is the leading indicator; the OOM is the lagging indicator.
• Run a du -sh /var/lib/prometheus/data/{head,wal} weekly and graph it. WAL size is the leading indicator for restart-time risk.

Verification

You should now be able to answer:

• What is one active series, and what does it cost in RAM on a Prometheus 2.55 host?
• How do you derive the bytes-per-series constant from the live metrics?
• Which three numbers (active series, samples per second, on-disk bytes) describe the steady state, and which metrics expose each?
• What does sample_limit do when breached, and why is that the failure shape you want?
• How do you size the platform cap from host memory and the measured bytes-per-series constant?

Quiz