Metric Cost Drivers

The Prometheus UI showed 5.2 million active series. The deployment had been running two years. The TSDB had grown to 1.1 TiB on disk. The WAL was at 12 GiB. Nobody had set a limit. The team learned the cost of metrics when a head-series OOM killed the server during a load test.

What metric cost is

A metric’s cost has three additive sub-costs:

• Head memory. Every active series lives in a head block in RAM. Each series carries a label-set, the most-recent value, and the most-recent timestamp. In Prometheus 2.55 the resident cost is roughly 3 KiB per series.
• WAL writes. Each scrape appends a record per series to the write-ahead log on local disk. Bytes-per-sample here is roughly 1.3 to 2.5 bytes after gzip and label compression.
• TSDB blocks. Every two hours the head is compacted into a 2 h block on disk and shipped to remote storage (Mimir, Thanos, S3). Disk cost is roughly the same per sample, with the addition of the index file.

The metric that ties them together is active series during the scrape window. Total resident cost is dominated by head memory; total on-disk cost is dominated by series count times retention in seconds times bytes-per-sample.

For a self-hosted Mimir / Cortex distributor the same model applies but the head lives in the ingester, and per-series memory is similar. For a vendor SaaS, the per-series cost is still on the invoice; it is just hidden behind a tier.

Why a sysadmin cares

Three failure shapes kill the metrics plane when cardinality runs away:

1. Head OOM. The most-recent-value store cannot fit a 5 M series head into 8 GiB of RSS. Prometheus crashes. Alertmanager misses alerts; remote-write backpressure stalls.
2. Slow queries. A histogram_quantile over a 5 M-series metric takes seconds. Dashboards rotate to “No data” because the query times out before the panel renders.
3. Ingester OOM (Mimir). Per-tenant ingesters serve queries and hold a chunk of the head. The largest tenant exceeds its per-tenant memory budget and the ingester is killed by its liveness probe.

In each shape the cost is bounded by a single limit:

• Head memory is bounded by scrape_samples_limit on each scrape target and by prometheus_tsdb_compactions and head trim behaviour on the server.
• Disk is bounded by storage.tsdb.retention.time, storage.tsdb.retention.size, and the remote-write endpoint capacity.
• Query latency is bounded by the cardinality of the label-set on every metric in the query, regardless of filters.

How the metric cost model works

The mental model is: scrape → label relabel → head append → compaction → block → query. Cardinality is added at each stage, and once added it is hard to remove.

   /metrics        labelmap, labeldrop, labelkeep
       |           (drops cardinality; replace; keep)
       v
  metric_relabel_configs  --->  cardinality-N0  (post-relabel)
       |
       v
  scrape_samples_limit     --->  cardinality-N1  (post-sample-cap)
       |
       v
  ingestion append         --->  cardinality-N2  (head memory + WAL)
       |
       v
  2h compaction            --->  cardinality-N2  (blocks on disk)
       |
       v
  recording rules          --->  cardinality-N3  (lower than N2)
       |
       v
  remote write             --->  cardinality-N3  (receiver holds the same)
       |
       v
  query (PromQL)           --->  O(active series in window)

The diagram’s takeaway is that a relabel rule below the scrape level is the only cheap place to drop cardinality. Once a series is in the head it costs memory and disk until retention expires. A recording rule that aggregates below the head is the only way to reduce per-query cost without losing the aggregated signal.

How to control metric cost

The right configuration isolates cardinality at the boundary between scraper and storage. A working prometheus.yml for a mid-size fleet reads as follows.

# File: /etc/prometheus/prometheus.yml
# Severity: CONFIGURATION (reload required with SIGHUP)

global:
  scrape_interval: 30s           # sample rate per series
  scrape_timeout: 10s            # cap on a single scrape
  external_labels:
    region: eu-west-1
    env: production

# Per-target sample cap rejects pathological responses early.
scrape_configs:
  - job_name: node
    sample_limit: 5000           # cap per scrape, refuses overflow
    target_limit: 200
    file_sd_configs:
      - files: ['/etc/prometheus/targets/node/*.json']
    metric_relabel_configs:
      # Keep only the labels we use in dashboards and alerts.
      - source_labels: [__name__]
        regex: 'node_(cpu|memory|disk|network).*'
        action: keep             # cardinality-N0 set explicitly
      # Drop high-cardinality dimensions before the head sees them.
      - source_labels: [hostname]
        regex: '.+'
        target_label: instance
        action: labelmap
      - regex: 'pod|container|namespace'
        action: labeldrop         # strip these from node metrics

  - job_name: app
    sample_limit: 2000
    metric_relabel_configs:
      # Per-request labels are the classic footgun. Drop before HEAD.
      - regex: 'request_id|session_id|trace_id'
        action: labeldrop
      # Cap on a specific label that has exploded in the past.
      - source_labels: [user_id]
        regex: '.+'
        action: drop              # remove the whole series, user_id alone is the key

# Recording rules reduce cardinality for query-time aggregates.
rule_files:
  - /etc/prometheus/rules/*.yaml

remote_write:
  - url: https://mimir.example/api/v1/push
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
    write_config:
      # Each remote_write batch respects a per-batch sample cap so
      # one backlogged tenant cannot starve another.
      max_samples_per_send: 2000

# File: /etc/prometheus/rules/api-aggregates.yaml
# Severity: CONFIGURATION (reload required with SIGHUP)
# Aggregates high-cardinality request metrics into bucket-level summaries.
groups:
  - name: api-aggregates
    interval: 30s
    rules:
      - record: api:request_latency_seconds:p99
        expr: |
          histogram_quantile(0.99,
            sum by (le, route, status) (rate(api_request_seconds_bucket[5m]))
          )
      - record: api:requests:rate1m
        expr: sum by (route, status) (rate(api_requests_total[1m]))
      - record: api:errors:rate1m
        expr: sum by (route) (rate(api_requests_total{status=~"5.."}[1m]))

The rule that does the work is metric_relabel_configs.action: keep on the __name__ of node metrics. From that point on the head sees exactly the metric names the dashboards and alerts are allowed to depend on, and any new metric that arrives under node_* is dropped before the head increments.

For remote-write receivers, the matching Mimir / Cortex limit is per-tenant ingestion_rate and max_global_series_per_metric on the distributor.

# File: mimir-distributor.yaml  (limits in Mimir 2.x)
limits:
  ingestion_rate: 20000           # samples per second per tenant
  ingestion_burst_size: 40000
  max_global_series_per_metric: 200000
  max_global_series_per_user: 5000000
  reject_older_samples: true
  creation_grace_period: 10m

How to validate metric cost

Three commands answer the questions that matter: how many series, who contributed them, and what the head looks like.

# Severity: READ-ONLY
# Active series, sampled every 15 s. A change > 5 percent per hour
# warrants investigation.
curl -s 'http://prometheus:9090/api/v1/query?query=prometheus_tsdb_head_series' \
  | jq -r '.data.result[0].value[1]'

illustrative: 1832447

# Severity: READ-ONLY
# Top ten series contributors by job. A single job that owns more
# than 30 percent of the head is the place to start.
curl -s 'http://prometheus:9090/api/v1/query?query=topk(10,%20count%20by%20(job)(up%3D%3D1))' \
  | jq '.data.result | sort_by(-.value[1])'

illustrative:
[
  {"metric":{"job":"node"},"value":[1,"3321"]},
  {"metric":{"job":"kube-state"},"value":[1,"2904"]},
  ...
]

# Severity: READ-ONLY
# Cardinality of one specific label across active series.
curl -s 'http://prometheus:9090/api/v1/query?query=count%20by%20(__name__)(count%20by%20(__name__%2C%20handler)(rate(node_cpu_seconds_total%5B5m%5D)))' \
  | jq .

The right response shape is a small list of well-known metric names with sensible cardinalities (10 to 1 000 each). A list with a single metric above 50 000 is the smoking gun.

# Severity: READ-ONLY
# Validate that the ruler loaded the recording-rule groups. Any
# failure here is a configuration issue, not a runtime issue.
promtool check rules /etc/prometheus/rules/*.yaml

illustrative:
Checking /etc/prometheus/rules/api-aggregates.yaml
  SUCCESS: 3 rules found

How it can fail

Six failure shapes for the metrics plane, each with a recognisable symptom.

1. The per-request label dropped into production. An engineer adds request_id as an exemplar-aware label. Head series triples in twelve hours; remote-write backpressure stalls; distributors return 429 sample limit reached to all tenants.
2. The aggregator scrape without a labeldrop. A misconfigured file_sd target list pulls every ServiceMonitor from every namespace. The same up series is scraped ten times with different instance labels and head series doubles.
3. The aggressive scrape interval. Someone halves the global scrape_interval from 30 s to 15 s. Sample ingestion doubles even though series count is unchanged. The byte cost on disk doubles; remote-write cost doubles.
4. The relabel regex that kept nothing. regex: '.*' with action: labeldrop was meant to drop a label but the label name was misspelled. No labels are dropped. Cardinality continues to grow.
5. The recording rule that aggregates nothing. sum by () on a metric with high cardinality means “one series per scrap,” not “one series globally”. The rule output is just as high-cardinality as the input.
6. The dropped WAL. A scratch volume filled; the WAL could not append; Prometheus logged out of order sample and the head refused to advance. Symptoms appeared as a flat-line up panel and missing alerts hours before the alert rule noticed.

How to troubleshoot runaway series

The diagnostic order is: count, attribute, decide.

Symptom (head_series jumped 30 percent in an hour)
   |
   +-- prometheus_tsdb_head_series total -- how far over budget?
   |
   +-- topk by job -- which job owns the growth?
   |     |
   |     +-- count by (__name__, job) -- which metric under it grew?
   |     |
   |     +-- count by (suspect_label) -- which label exploded?
   |
   +-- Decide: keep, relabel, drop, or rule?
   |     |
   |     |-- keep   -- the metric has a dashboard alert; relabel the
   |     |            bad labels away, leave the rest
   |     |-- rule   -- the metric is high-cardinality but is queried
   |     |            in aggregate; add a recording rule
   |     |-- drop   -- the metric has no consumer; labeldrop or drop
   |
   +-- Verify: did prometheus_tsdb_head_series fall?
   +-- Document: cost platform change log
   |
Root cause

The decision point is non-trivial. “Drop” is correct for series with no consumer, but is catastrophic for series that an alert depends on. Rule before drop. Relabel before rule. Document before any change touches the head.

Security implications

A label set can leak. A metric labelled with a user email, an order ID, or a stack-trace path exposes business data to anyone who can read the Prometheus API. The fix is at the relabel layer, not at the access-control layer: the sysadmin never lets the data into the head. Per-scrape target ACLs are also essential. A scrape config that targets service-discovery:consul on an untrusted network can pull labels from external sources the sysadmin does not control. Network ACLs and mutual TLS to the SD endpoint are the right control surface; a Prometheus that scrapes the world is a Prometheus whose cardinality you cannot budget.

Performance implications

Memory and query latency dominate. A 5 M active series head at roughly 3 KiB per series is 15 GiB of RSS before query and compaction. RAM is the ceiling for head size; the next ceiling is compaction throughput, which is governed by storage.tsdb.max-block-chunk-segment-size and the rate of new blocks. The third ceiling is the query engine, which materialises every series in a sum by () without a label. Per-label-set cardinality on the largest queried metric is the constraint that determines whether a dashboard renders.

The discipline is to keep memory headroom, to keep compaction ahead of the scrape interval, and to keep query cardinalities bounded by relabel rules. Recording rules reduce the latter but do nothing for the first two.

Production guidance

• Set sample_limit on every scrape target to roughly twice the steady-state sample count of that target class. A ceiling that never trips is a config comment.
• Use metric_relabel_configs.action: keep with a __name__ regex to declare the metrics the platform is allowed to ingest per job. Anything else is dropped before the head.
• Apply recording rules at the metric level, not at the label level. The rule’s output metric should have a lower cardinality than its input.
• Watch prometheus_tsdb_head_series and alert at roughly 80 percent of budget. The shape that catches runs away is the one that is “ramping gently” — the doubling over a week.
• Document every relabel change in a runbook. A label dropped by accident can break an alert that has shipped for years.

Verification

You should now be able to answer:

• What three sub-costs make up the cost of a single metric, and which has the hardest ceiling?
• What is cardinality, in terms of labels and values, and what is the Prometheus 2.55 per-series memory cost?
• Where in prometheus.yml does the cardinality control belong, and why does moving it later break the budget?
• How does a recording rule reduce query cost, and why does sum by () not do it on its own?
• What is the right diagnostic order when prometheus_tsdb_head_series climbs 30 percent in an hour?

Quiz