Break/Fix: Cardinality Explosion · Break/Fix · Observability

Reported symptoms

Prometheus was OOM-killed at 09:40 on Tuesday and again at 15:10 on Thursday. Both times somebody was looking at a 30-day panel when it went. Between the kills, resident memory does not spike - it ramps, from 6.2 GiB nine days ago to 14 GiB now, with no step anywhere on the graph.

The data directory has grown about fourfold over the same nine days. Retention has not changed. The scrape interval has not changed. The target count has not changed.

Three other things are wrong, and none of them sounds like the same problem:

AppHighErrorRate has a for: 10m gate. On Thursday afternoon the error rate was above threshold for over an hour and the alert never fired. The same rule fires perfectly well at night, which is why the SRE team has it filed as flaky rather than broken.
The deploy dashboard’s 24-hour increase() panel reports about a third of the request count the application’s own access logs record. It has been wrong for over a week and was assumed to be a units mistake.
Long-range queries are slow to the point of timing out. Anything under a few hours is fine.

And the two alerts that exist for exactly this failure have both stayed quiet. Head series is up around twenty percent - real, but nowhere near the five-million ceiling and nowhere near three times the six-hour baseline that the step detector watches for.

The application team has already ruled themselves out, and they are right to. No metric was added, no label was added to any instrumentation, and the app repository has been quiet for six weeks.

Evidence provided

Read-only / Safethe level - up from ~2.1M nine days ago, and not the story

$ curl -s 'http://prometheus:9090/api/v1/query?query=prometheus_tsdb_head_series' \
| jq -r '.data.result[0].value[1]'

Illustrative output

Read-only / Safeseries created per second - the thirty-day baseline is 0.7

$ curl -s --data-urlencode 'query=rate(prometheus_tsdb_head_series_created_total[1h])' \
http://prometheus:9090/api/v1/query | jq -r '.data.result[0].value[1]'

48.2

Illustrative output

Read-only / Safedistinct values per label name, straight from the TSDB

$ curl -s http://prometheus:9090/api/v1/status/tsdb \
| jq '.data.labelValueCountByLabelName'

[
{ "name": "build",    "value": 108 },
{ "name": "instance", "value": 40 },
{ "name": "job",      "value": 11 },
{ "name": "status",   "value": 7 },
{ "name": "method",   "value": 5 }
]

Illustrative output

Read-only / Safethe alert series itself, and what it is carrying

$ curl -s --data-urlencode 'query=ALERTS{alertname="AppHighErrorRate"}' \
http://prometheus:9090/api/v1/query | jq -c '.data.result[].metric'

{"__name__":"ALERTS","alertname":"AppHighErrorRate","alertstate":"pending","build":"9a2c1f","instance":"app-07:9100","job":"app","severity":"critical"}

Illustrative output

The app job’s discovery, as the CD pipeline writes it every deploy:

[
  {
    "targets": ["app-01:9100", "app-02:9100", "app-03:9100"],
    "labels": {
      "job": "app",
      "env": "prod",
      "build": "9a2c1f"
    }
  }
]

And the rule that has not been firing:

- alert: AppHighErrorRate
  expr: |
    sum without (status) (rate(app_http_requests_total{status=~"5.."}[5m]))
      /
    sum without (status) (rate(app_http_requests_total[5m]))
      > 0.05
  for: 10m
  labels:
    severity: critical

For scale: the app job has 40 targets, each exposing roughly 3,000 series, and the pipeline deploys about a dozen times on a weekday.

Work the evidence before reading on

The two alerts written for cardinality incidents are both correct and both silent. Take that as information rather than as a fault.

The head series count rose by twenty percent. The series creation rate rose by a factor of about seventy. What is the difference between those two quantities, and which one does a level alert measure?
build has 108 distinct values and instance has 40. The estate has 40 app targets. What is 108, and what happened roughly 108 times in the last nine days?
The top-metric-families query returns a flat list. Under what cause would every metric family from one job grow by the same factor, and under what cause would one family dominate?
Disk grew fourfold but the chunks files barely changed; the growth is in index. What does a block’s index contain that its chunks do not?
The ALERTS series carries a build label. Where did the rule get it from, and what happens to a for: clock when the series it is counting against is not the same series it was a minute ago?
Both OOMs happened while a 30-day panel was open. Why would query range matter here when the live series count is only twenty percent up?

Before continuing: which single change nine days ago is upstream of all six of those observations?

Root cause

A target label is attached to everything the target exposes

build is not a label on a metric. It is a label on a target, written into the file service-discovery output by the CD pipeline so the deploy dashboard could annotate releases. Target labels are merged into every sample the target returns, so build rides on all 3,000 series each app instance exposes, not on one.

A time series is identified by its metric name together with the whole of its label set. Changing one label value therefore does not annotate an existing series - it produces a different series. Every deploy retires the whole generation and mints a fresh one: 40 targets times 3,000 series, or about 120,000 series abandoned and 120,000 created, twelve times a weekday. Nine days of that is the 108 distinct build values the TSDB is reporting.

That single mechanism is upstream of everything on the list. Every family from that job grew by the same factor because the multiplier is applied at the target, not at the metric - which is exactly why the top-metric-families query that the cardinality runbook opens with returned a flat, useless list.

The level did not move, so the level alerts could not see it

This is the part worth internalising. A retired series receives a staleness marker, stops being appended to, and is dropped from the head block at the next head truncation. The instantaneous series count therefore only ever holds the generations created inside the current truncation window - a handful, not 108 - so prometheus_tsdb_head_series rose by twenty percent and sat there. Both alerts on that gauge behaved exactly as designed and neither had anything to fire on.

The quantity that moved is churn: series created per unit time, which prometheus_tsdb_head_series_created_total counts and nothing in this estate was watching. Churn is its own axis. It is what drives WAL volume, the size of each block’s index, and compaction cost, and it can rise by two orders of magnitude while the level stays almost flat.

That is also why the disk growth is in index and not in chunks. Every generation minted inside a two-hour window gets index entries in that window’s block - series references, postings for each label pair, symbol-table entries

while the samples themselves are unchanged in volume. The index carries the cost of how many series existed; the chunks carry the cost of how many samples arrived, and only the first of those changed.

And it is why a 30-day panel is what kills the process. A long-range query has to merge every generation in every block it touches. Ninety days of normal operation would have been about 120,000 app series to merge; nine days of this is over ten million. The live series count is a poor predictor of that cost, which is why the process looks survivable right up until somebody widens a time picker.

The two symptoms that looked like application faults

A counter that changes identity restarts at zero under its new label set. The old series ends, the new one begins, and nothing connects them - so a increase() over 24 hours sees only the fragment of the window that its current series existed for. That is the deploy dashboard undercounting by roughly two thirds, and it is arithmetic rather than a units mistake.

The alert is the same mechanism one level up. The rule aggregates with sum without (status), which drops status and keeps everything else - including build. The output series therefore carries the build SHA, so at every deploy the alert is evaluating a new series and its for: 10m clock starts again from zero. On a quiet night, with no deploys, the clock runs its full ten minutes and the alert fires. On a release afternoon, with a deploy every forty minutes or less, it never gets there. The rule is not flaky. It is being reset, by the pipeline, on a schedule.

Level and churn are different questions and need different alerts

Almost every cardinality runbook opens with prometheus_tsdb_head_series and then with the top metric families by series count. Both of those questions are about level: how many series exist right now, and which metric owns them. They are the right questions for the classic incident, where an unbounded label such as user_id puts millions of series into the head simultaneously.

They are the wrong questions for a churn incident, and they fail in the most expensive way - by returning confident, correct, uninformative answers. The level is only mildly elevated. The metric families are all equally grown. An on-call engineer who trusts the runbook’s shape concludes there is no cardinality problem and goes looking at the application.

The question that separates them costs one query: rate(prometheus_tsdb_head_series_created_total[1h]), read against its own history. A level that is flat with churn seventy times baseline means series are being replaced rather than accumulated, and that points at label values changing on a schedule - deploys, rollouts, restarts, a generator rewriting a target file - rather than at an unbounded dimension.

Resolution

Protect the process before you fix the cause. The OOMs come from long-range queries against bloated block indexes, so cap or shorten the 30-day panels first; that is the difference between working the incident and working it between restarts.
Tell the SRE team that AppHighErrorRate cannot fire during a release, in writing, and arrange cover. That gap has existed for nine days and it does not close until the rule is changed.
Apply the brake if the pipeline cannot be changed immediately: action: labeldrop on build in the app job's metric_relabel_configs, which strips the label after the scrape and before ingestion. Validate with promtool check config, then reload. No further generations are minted from the next scrape onward.
Fix the cause at source. Stop the CD pipeline writing build into /etc/prometheus/file_sd/app.json. The relabel drop is a brake on a runaway; the target file is where the runaway starts.
Ship the replacement in the same change, not after it. Expose app_build_info{build="..."} 1 from the application - one series per target, the convention node_exporter_build_info and prometheus_build_info already follow - and re-point the deploy annotation at it. A fix that removes the capability somebody asked for gets reverted by whoever asked.
Correct the alert rule to aggregate with by over the labels the alert is actually about, so its identity is a decision the author made rather than whatever arrives from upstream. Test it with promtool test rules before it goes near production.
Pick the moment. Removing the label changes series identity one final time: every rate and increase panel takes one more notch, every without-based rule resets its clock once more, and any recording rule that carried build produces a new output series. Outside a release window, with the team told in advance.
Do not restart Prometheus to reclaim the memory. The RSS descent after the fix is gradual - the head truncates, blocks compact - and a restart adds a WAL replay under exactly the memory pressure you are trying to relieve.
Open the follow-up before closing the incident: the churn alert, and the cardinality budget entry for target labels. The brake and the source fix both leave the estate one review away from the next generator that decides a label would be useful.

Verification

Churn is back to baseline and stays there through a deploy. rate(prometheus_tsdb_head_series_created_total[1h]) returning to single digits proves the brake works; watching one deliberate deploy go past with no spike proves the cause is gone. Only the second is the real test.
The label is absent rather than reduced. /api/v1/status/tsdb must no longer list build under labelValueCountByLabelName, and count(count by (build) (app_http_requests_total)) must return no data. A smaller number means something is still emitting it.
The derived series are clean. ALERTS{alertname="AppHighErrorRate"} no longer carries build, and the pending clock survives a release - hold the rule in pending across a deploy and watch it reach firing rather than reset.
The counters are continuous again. Run the 24-hour increase() over a window that contains several deploys and reconcile it against the application access logs; agreement within a few percent is the pass, and a third of the truth is the failure you started with.
The replacement carries the capability. app_build_info exists, has exactly one series per target, and the deploy annotation on the dashboard reads from it.
The platform is settling, not settled. RSS descending over hours rather than snapping back is the expected shape. Block index sizes returning to their previous scale applies only to blocks written from now on.
Long-range queries are still slow, and you have said so. The blocks already on disk keep their bloated indexes until retention ages them out; telling the reporting team the dashboards are fixed today is a claim the disk will contradict.
The churn alert fires when it should. Point it at the nine days of history you have just lived through and confirm it would have tripped on day one. An alert that cannot detect the incident that motivated it is decoration.

Prevention

Review target labels as job-wide changes, because that is what they are. A label written into a discovery file or added by relabel_configs lands on every series every target in that job exposes. The diff is one line; the blast radius is the job.
Keep label values stable for unchanged hosts. A generator that rewrites a label on a target which has not otherwise changed is manufacturing churn on a schedule, and the schedule is usually the deploy cadence - which is to say, it is worst exactly when the alerts matter most.
Alert on churn as well as level. A rule on rate(prometheus_tsdb_head_series_created_total[1h]) compared against its own baseline catches this whole class. The level alerts cannot, and this incident is nine days of proof.
Put identity in an info metric. app_build_info{build="..."} 1 is one series per target, joined at query time, and it is what every exporter in the estate already does with its own version. Build, release, commit and version belong there and nowhere else.
Prefer by over without in rules that feed alerts. without inherits whatever labels arrive from upstream, which makes the alert’s identity - and therefore its for: clock - a property of somebody else’s config.
Add distinct-value counts for target labels to the cardinality budget review. count(count by (build) (app_http_requests_total)) returning 108 against an inventory of 40 targets is one query and an obvious answer; nine days of dashboards had it hidden in plain sight.

What if the build SHA had gone on one metric instead of on the target?

Then this would have been a much smaller incident, and an obvious one. A build label on a single counter multiplies that counter’s series by the number of builds and leaves everything else alone - so the top-metric-families query would have named the offender immediately, the runbook’s first step would have worked, and the fix would have been a labeldrop on one job with no derived-series consequences.

The failure has a mirror image that is worth holding next to it, because the two look identical in a graph of memory and opposite in every diagnostic. A classic cardinality explosion - user_id on a request counter - puts millions of series into the head simultaneously. The level goes up and stays up, prometheus_tsdb_head_series triples inside a scrape interval, both standard alerts fire, one metric family dominates the top-N, and churn is unremarkable because the series are accumulating rather than rotating. Level high and churn flat means an unbounded dimension. Level flat and churn high means an identity that keeps changing. Reading which of the two you have, before choosing a query, is most of the diagnosis.

There is a third variant that catches people during migrations: a mass metric rename, or a fleet-wide relabel change applied at once. That is a single, very large churn event rather than a repeating one - a visible bump in head series and WAL traffic for a few hours, then quiet. If the churn graph is a step that decays rather than a sawtooth, look for one change rather than a schedule.

Why this matters

The cardinality runbook most teams have written measures level, because the incident most teams have had was a level incident. That runbook is correct and it would have found nothing here for nine days while the platform slowly died, because the quantity that moved was never on it.

The transferable habit is to hold two numbers rather than one: how many series exist, and how fast series are being created. They answer different questions, they fail in different directions, and a label that is completely safe by the first measure - build has 108 values, which is nothing - can be ruinous by the second. Anything that stamps a value onto every series a target emits, and changes that value on a schedule, is a churn source. Deploys are the common one. Rollouts, restarts, IP reassignments and helpfully regenerated inventory files are the rest.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

A target label is attached to everything the target exposes

The level did not move, so the level alerts could not see it

The two symptoms that looked like application faults

Resolution

Verification

Prevention