Break/Fix: Prometheus OOM

Reported symptoms

Prometheus has been OOM-killed between three and six times a day for about a month. Always on a weekday, always between 07:00 and 22:00, never overnight and never at a weekend.

The memory graph does not look like an out-of-memory problem. Resident memory sits at 9 GiB against a 24 GiB cgroup limit and has sat there for months. The kills are vertical: from 9 GiB to the limit in under forty seconds, with no ramp before them and no elevated plateau after. The host has 64 GiB and shows around 40 GiB free at the moment of each kill, which is why the first week of investigation went into the host and found nothing.

The cardinality runbook comes back clean at every step. Head series is flat at about 1.4M and has been for four months. Ingestion is flat at about 48,000 samples per second. No metric family dominates. Series creation rate is unremarkable. Nothing has been added to the scrape config since spring.

Three other things are wrong, and none of them sounds like the same problem:

• Two alerts that should have fired during a real incident last Thursday did not fire at all. Nobody has explained it and the incident review has the question open.
• Users on unrelated dashboards see query timed out during the events.
• Two panels on the NOC capacity dashboard have shown query processing would load too many samples into memory for a month. They are in the backlog as broken panels.

The only change anywhere in the window is a dashboard edit. Its pull request reads, in full, “add instance selector, tidy panel titles”.

Evidence provided

$ journalctl -u prometheus --since '-1h' | grep -A4 'last run'

level=info msg="These queries didn't finish in prometheus' last run:"
query="node_memory_MemAvailable_bytes{instance=~".*"}" timestamp_sec=1755231043
query="node_memory_MemAvailable_bytes{instance=~".*"}" timestamp_sec=1755231043
query="node_load15{instance=~".*"}" timestamp_sec=1755231044
query="node_network_receive_bytes_total{instance=~".*"}" timestamp_sec=1755231044

$ curl -s --data-urlencode 'query=prometheus_engine_queries' \
http://prometheus:9090/api/v1/query | jq -r '.data.result[0].value[1]'

20

$ curl -s --data-urlencode 'query=count(node_memory_MemAvailable_bytes)' \
http://prometheus:9090/api/v1/query | jq -r '.data.result[0].value[1]'

260

The request Grafana is sending, taken from the access log during a refresh:

POST /api/v1/query_range
  query = node_memory_MemAvailable_bytes{instance=~".*"}
  start = 1752639043
  end   = 1755231043
  step  = 15

The flags the unit runs with:

# /etc/default/prometheus
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --query.max-concurrency=20 \
      --query.timeout=2m \
      --query.max-samples=50000000"

And the rule-evaluation health across the same events:

prometheus_rule_group_iterations_missed_total   # climbs during every event
prometheus_rule_group_last_duration_seconds     # platform group well past its 30s interval

The dashboard diff turns Multi-value and Include All on for the instance variable, and removes the Min interval field from five panels. The dashboard default range is thirty days. Four kiosk displays in the NOC hold it open with a thirty-second auto-refresh.

Work the evidence before reading on

The cardinality runbook is correct, was executed correctly, and found nothing. Treat that as information rather than as a dead end.

1. Head series is flat and the kills are cliffs of under forty seconds. What memory cost in Prometheus can appear and disappear on that timescale, and what cost cannot?
2. The host has 40 GiB free at the moment of each kill. What killed the process, and what does that tell you about which accounting boundary was crossed?
3. Work the arithmetic for one panel: 260 series, thirty days, a fifteen-second step. How many samples is that, and how does the number compare with --query.max-samples=50000000?
4. Now do it again for twenty of them. Which flag bounds that total?
5. Two panels on the same dashboard are rejected outright while three are served. What is different about the two, and why is being rejected the cheaper outcome?
6. prometheus_rule_group_iterations_missed_total climbs during each event. What do rule evaluations and dashboard queries have in common, and what does a missed iteration mean for an alert with a for: clause?

Before continuing: name the product of two configured numbers that is the real bound on this process’s query memory, and say why neither number on its own ever looked wrong.

Root cause

Every query was legal, and twenty of them at once were not

Thirty days at a fifteen-second step is 172,800 points per series. The instance variable expands All to the regex .*, so the panel selects all 260 series that node_memory_MemAvailable_bytes has. That is about 45 million samples for one panel - under --query.max-samples=50000000, so Prometheus accepts it, serves it, and is behaving exactly as configured.

max-samples is a ceiling on a single query. There is no aggregate ceiling. The real bound on resident query memory is therefore max-samples multiplied by max-concurrency: fifty million times twenty, or a billion samples, which at roughly sixteen bytes per decoded sample is far more than the 24 GiB the process is allowed. Four kiosk displays refreshing a five-panel dashboard every thirty seconds is enough to fill all twenty slots with individually legal queries, and the process crosses its limit in tens of seconds.

That is the shape of the graph. A cliff rather than a ramp, because query memory is allocated and released on the timescale of a query rather than accumulating; nothing in the head, the WAL or the block layer moves at all.

The rejected panels were the diagnosis, printed on the screen

Two of the five panels select per-mountpoint series rather than per-host ones, so they cross fifty million samples and are rejected before any of it is loaded. query processing would load too many samples into memory is not a broken panel. It is Prometheus stating the cause of the incident, on the same dashboard, in the same room, for a month.

Being rejected is the cheap outcome and it is worth understanding why: the engine counts the samples the query will touch and refuses before allocating, so a rejected query costs almost nothing. The panels that killed the process are the ones that came in just under the ceiling and were therefore served.

The alerting degradation is the part that mattered most

Rule evaluation and dashboard queries share the query scheduler. When twenty slots are held by thirty-second range queries, rule groups queue behind them, prometheus_rule_group_last_duration_seconds rises past the group interval, and prometheus_rule_group_iterations_missed_total starts counting.

An alert with a for: clause needs consecutive evaluations to reach firing. A group that misses ticks during exactly the window when something is going wrong is a group whose alerts may never elapse - which is the answer to the open question from last Thursday’s incident review. The dashboard did not only kill Prometheus; while it was doing so, it disabled the alerting.

Resolution

1. Take the pressure off first, on the same day. Set the kiosk auto-refresh to five minutes or turn it off, and tell the NOC why. Six kills a day is not a state to do careful work in.
2. Say out loud that alerting has been degraded for a month. prometheus_rule_group_iterations_missed_total has the periods; the on-call rotation and whoever ran last Thursday needs them in writing, because some alerts could not have fired.
3. Restore a Min interval on the five panels so the step scales with the range instead of being pinned at fifteen seconds. A thirty-day panel asking for a step in the hundreds of seconds cuts the sample count by more than an order of magnitude on its own.
4. Remove Include All from the 260-value instance variable, or cap what it expands to. All is a regex that matches everything, and it is a 260-fold multiplier on every panel that uses the variable.
5. Move the panels that genuinely need thirty days of the whole fleet onto a recording rule. That turns tens of millions of samples into tens of thousands and stays correct as the fleet grows, which neither of the previous two steps does.
6. Fix the two rejected panels in the same change. They are the same bug, they have been carrying the diagnosis for a month, and leaving them broken guarantees the next person also treats the message as noise.
7. Bound the product, because this will not be the last dashboard anyone writes. Lower --query.max-concurrency until max-samples times the gate, at roughly sixteen bytes a sample, fits inside the memory limit above the 9 GiB baseline.
8. Understand what the smaller gate costs before shipping it: queries do not get cheaper, they queue, and a queued query returns query timed out at the two-minute mark. That is a far better failure than a dead process and it is still a degradation people should hear about first.
9. Treat lowering --query.max-samples as the sharper brake and pick the number from measurement. It will also reject legitimate long-range work, and it can reject a rule evaluation.
10. Do not add memory as the fix. The product of the two flags is a billion samples; whatever ceiling you raise it to, that arithmetic will find again, and a larger heap lengthens the WAL replay after the next kill.

Verification

1. Reproduce it deliberately, in a window, with somebody watching. Open the dashboard on its thirty-day default and watch prometheus_engine_queries and process_resident_memory_bytes together. Testing it on a six-hour range tests the case that was never broken.
2. The gate does not saturate and resident memory moves by hundreds of megabytes rather than gigabytes. Both, not either.
3. The step really changed. Read data/queries.active while the dashboard is open, or read the step parameter off the request, and confirm a thirty-day panel is no longer asking for fifteen seconds. The panel setting is the intent; the request is the fact.
4. The two rejected panels render. If they still return the max-samples error, the fix did not reach them and they are still telling you so.
5. Alerting recovered. prometheus_rule_group_iterations_missed_total stops climbing, and one group that was missing ticks evaluates on schedule through a full working day. This is the check with real consequences attached.
6. The brake is wired up. Run one query deliberately above the max-samples ceiling and confirm it is rejected rather than fatal. A limit nobody has watched reject anything is a limit nobody knows works.
7. The 9 GiB baseline is unchanged. If it fell, something was altered that was not meant to be, and you should find out what before closing.
8. Frequency is the measurement. A full week with no kills, including a month-end, when the capacity dashboard gets its heaviest use.

Prevention

• Write the product down. max-samples times max-concurrency, times the per-sample cost, against the memory limit. That product is the only bound on resident query memory that exists, and with the defaults it comfortably exceeds most limits - which means the arithmetic can tell you the process is one dashboard away from a kill before the dashboard is written.
• Make samples-touched part of dashboard review: series multiplied by range divided by step, worked for the panel default and again for the widest range a user can pick.
• Never pin a step. A panel with a fixed step has a cost that scales with the time picker, and the time picker belongs to whoever opens the dashboard.
• Treat Include All on a high-cardinality variable as a multiplier, not a convenience. It expands to a regex that matches everything.
• Use recording rules as a memory control and not only as a latency one. A wide panel reading a pre-aggregated series is constant-cost however large the fleet grows.
• Alert on prometheus_engine_queries reaching prometheus_engine_queries_concurrent_max. A saturated gate precedes the kill by tens of seconds and is the earliest signal on offer.
• Alert on prometheus_rule_group_iterations_missed_total. This failure mode degrades alerting quietly, and an alerting platform that has silently stopped alerting is the worst state it can occupy.
• Read data/queries.active after any unexplained restart. Prometheus maintains it for precisely this and reports it on the next start.