Runbook: Investigate Prometheus High Memory

“Prometheus memory is climbing” is a symptom, not a diagnosis. At least four different incidents produce it, they have four different fixes, and exactly one of them is improved by adding RAM. Two of the four are not incidents at all.

The good news is that splitting them is one command. The bad news is that the reflex - restart it, or give it more memory - is right for one branch, actively harmful for two of the others, and merely expensive for the fourth.

When this runbook applies, and when it does not

It applies when the Prometheus process is consuming more memory than its budget allows, or is climbing towards that point.

It does not apply when:

The host is short of memory and Prometheus is not the largest consumer. That is a host incident. Prometheus may be its victim rather than its cause, and the fix is on the host, not in prometheus.yml.
Prometheus is already down and will not start. A process that exits during startup is a different runbook. This one assumes a running Prometheus you can still query, because every diagnostic here is an HTTP call to it.
The alert is on the host memory panel rather than on the process. Used memory on a Linux host is not a fault condition; a host at 92 percent used with healthy available memory is a page cache working correctly. Confirm the process before opening this runbook at all.
The memory pressure is on a remote-write receiver, a long-term store or a querier in front of Prometheus. Those have their own memory models and their own budgets; the head-series relationship this runbook is built on does not describe them.

Ask the second question first

Before diagnosing anything, answer a different question: how long do I have?

Headroom decides the order of everything else. With sixty percent of the ceiling free and a gentle slope, there is time to investigate properly and arrive at the real cause. With ten percent free and RSS rising, the process will be killed part-way through the investigation, and you will have paid for the investigation and received nothing - plus a replay, plus a gap.

That is not an argument for acting blindly. It is an argument for knowing which mode you are in, because the two modes have different correct first moves.

Read-only / Safehow much room is left, and how fast it is going

PROM=http://localhost:9090

# The process, not the host.
curl -s "$PROM/metrics" \
| grep -E '^(prometheus_tsdb_head_series|process_resident_memory_bytes|process_start_time_seconds) '

# The ceiling that actually applies. Prometheus does not limit itself.
cat /sys/fs/cgroup/memory.max 2>/dev/null \
|| systemctl show prometheus -p MemoryMax

# The slope, over the last hour.
curl -sG "$PROM/api/v1/query" --data-urlencode \
'query=process_resident_memory_bytes - (process_resident_memory_bytes offset 1h)' \
| jq -r '.data.result[].value[1]'

prometheus_tsdb_head_series 1.84e+06
process_resident_memory_bytes 1.1817e+10
process_start_time_seconds 1.755e+09
16106127360
2214592512

Illustrative output

Read the host separately and read it correctly: a host at 92 percent used with healthy MemAvailable is a page cache doing its job. The number that predicts an OOM is available memory and memory pressure, not used memory. A dashboard that reports “used” will show this incident as fine right up to the kill.

The branch: two numbers, read at the same moment

`prometheus_tsdb_head_series`	`process_resident_memory_bytes`	What it is	Where to go
Climbing	Climbing	A cardinality incident	The high-cardinality runbook, not this one
Flat	Climbing	The query path	Branch B below
Any	High, just after a restart, falling	WAL replay	Wait. Do not restart
Flat	Stepped up once, then flat	A change	Branch C below
Slowly rising over weeks	Tracking it	Drift	A capacity review, not a page

Reading these an hour apart makes the relationship unknowable, which is why they come from a single call. “Head series was high yesterday and memory is high today” is two facts and no diagnosis.

Branch A - head series climbing with RSS

This is cardinality, and it has its own runbook because the ordering matters more than the commands: the inflow has to be stopped before the memory is freed, or the head simply refills on the same curve after the restart.

The only thing to add here is the handoff. Do not restart to reclaim memory while the offending label is still arriving. A restart in that state buys exactly one WAL replay of time and makes the following hour harder, because the replay consumes memory of its own and delays the first scrape cycle.

Go to the high-cardinality runbook and come back when the inflow is confirmed stopped.

Branch B - head series flat, RSS climbing

The series count is stable, so nothing new is being stored. The memory is being spent evaluating something.

Read-only / Safethe query path

PROM=http://localhost:9090

curl -s "$PROM/metrics" \
| grep -E '^prometheus_engine_queries(_concurrent_max)? '

curl -sG "$PROM/api/v1/query" --data-urlencode \
'query=histogram_quantile(0.99, sum by (le) (rate(prometheus_engine_query_duration_seconds_bucket[5m])))' \
| jq -r '.data.result[].value[1]'

# What is running RIGHT NOW. The path comes from the running flags,
# not from the unit file.
TSDB=$(curl -s "$PROM/api/v1/status/flags" | jq -r '.data["storage.tsdb.path"]')
cat "$TSDB/queries.active"

prometheus_engine_queries 20
prometheus_engine_queries_concurrent_max 20
14.8

Illustrative output

Concurrent queries pinned at the maximum is the signature. Each one holds its working set in memory until it completes or is killed, so the memory cost is the sum of the in-flight queries rather than the cost of the worst one.

queries.active is the piece operators usually do not know exists. It is the only place a running Prometheus names the queries it is currently working on, and it turns “something is expensive” into “this expression, from this dashboard”.

Name the source before killing anything. A query issued by a dashboard on a thirty-second refresh comes back thirty seconds later, and killing it in a loop while the dashboard is open is a way to spend an hour learning nothing.

Branch C - it stepped up once and stayed

A single step with a flat line either side is a change, not a leak. The step time is the search key.

Read-only / Safewhat the running process actually believes

PROM=http://localhost:9090

curl -s "$PROM/api/v1/status/flags" | jq '.data | {
retention: .["storage.tsdb.retention.time"],
path: .["storage.tsdb.path"],
concurrency: .["query.max-concurrency"],
timeout: .["query.timeout"]
}'

curl -s "$PROM/api/v1/status/config" | head -40

This reads the running process rather than the unit file or the repository. If the two disagree, that disagreement is itself the finding - and it is a common one after a change that was applied by editing a file nobody reloaded.

A step that lines up with a new scrape job, a new rule group or a raised concurrency limit is a capacity change. It is legitimate, it is not a defect, and the correct response is a budget decision made in daylight rather than a mitigation applied at 03:00.

Branch D - it has been growing for weeks

A line that has been rising gently since the last capacity review, with head series and RSS tracking each other, is drift. Nothing broke. The platform has been absorbing new targets, new services and new rules, one small increment at a time, and it has walked into its ceiling.

This is the branch where the correct response is to not treat it as an incident. There is no offender to name and no mitigation to apply; a relabel rule written at 03:00 against drift removes a label somebody chose on purpose. What it needs is the arithmetic, in daylight: current active series, the measured bytes per series on this instance rather than the planning range, the growth rate per week, and the date the projection crosses the ceiling.

The reason to be strict about this is the page. An instance that pages every few weeks for drift teaches the rotation that this alert does not mean anything, and the rotation is right - which is a problem the first time the same alert fires for Branch A.

Choosing the holding action

The actions are not interchangeable, and the table is the reason.

Action	Reversible?	What it costs	When it is right
Kill one in-flight query	n/a	Nothing; it will be re-issued	Branch B, immediately
Lower `--query.max-concurrency`	Yes, via another restart	A restart now and a restart to revert	Branch B, when the source cannot be stopped
Raise the memory ceiling	On paper only	Nothing now; an OOM later if growth is unbounded	Only when the demand is intentional
Drop or limit a scrape job	Yes, on reload	That job has no data for the window, ever	When one job is the whole cause
Restart Prometheus	No	A replay-length gap, and the replay’s own memory	Last, and only when priced
Lower retention	Yes	Deleted history, and zero memory recovered	Never, for this symptom

Raising the ceiling deserves its own note, because it is the action this runbook is most often read as forbidding. It is not. When the series are intentional - a real service, deliberately instrumented, inside a budget somebody agreed to - then the instance is simply undersized and more memory is the honest answer. The test is whether the demand is bounded. Intentional growth has a ceiling you can name; an unbounded label does not, and against that, more memory buys time proportional to the growth rate and nothing else.

Pricing the restart

If the answer really is a restart, price it before taking it. The two costs are the replay time and the alerting gap, and both are measurable in advance.

Read-only / Safewhat the restart will cost

PROM=http://localhost:9090
TSDB=$(curl -s "$PROM/api/v1/status/flags" | jq -r '.data["storage.tsdb.path"]')

# The WAL is what has to be replayed. Bigger WAL, longer blind window.
du -sh "$TSDB/wal"

# Is anything else carrying alerting while this one is down?
curl -s "$PROM/api/v1/alertmanagers" | jq -r '.data.activeAlertmanagers[].url'

4.7G	/var/lib/prometheus/data/wal

Illustrative output

A large WAL is itself a finding: high series churn produces a big WAL, and churn is what makes restarts expensive. That is why the same mitigation costs five minutes on one instance and forty on another, and why “just restart it” is not a portable instruction.

Then answer the question the restart raises: who pages while this instance is replaying? An HA replica, a remote-write receiver, or nobody. If the answer is nobody, say so out loud before doing it rather than discovering it afterwards.

Why this matters

This runbook exists because the obvious response to a memory alert is wrong more often than it is right. Adding memory to a cardinality incident buys time proportional to the growth rate and fixes nothing. Restarting during a WAL replay extends the outage. Cutting retention deletes data and recovers no RAM. Each of those is a reasonable-sounding action taken by a competent person under pressure, and each makes the incident longer.

What prevents all three is the same ninety seconds: read head series and resident memory together, read the headroom and the slope, and check whether the process has just started. Those four numbers put you in the right branch, and the branch determines everything else.

The reason it is worth being this careful is what Prometheus is. When it is degraded, alert rules evaluate late or not at all, and the platform’s ability to detect any other problem falls at the same moment. An incident here is never only an incident here.

Common patterns

Symptom	Likely cause	Resolution
Head series and RSS climbing together	Cardinality	The high-cardinality runbook; stop the inflow first
RSS high and falling shortly after a start	WAL replay in progress	Wait; measure the WAL to know how long
RSS climbing, head series flat, queries pinned at max	An expensive query, usually a dashboard	Name it from `queries.active`, then talk to its owner
RSS stepped once and stayed	A config, flag or rule change	Diff `/api/v1/status/flags` and the config against the repository
Memory falls fast then sits flat and high for an hour	The mitigation reached the head but not the real consumer	Re-diagnose; the branch was wrong
Recovers, then recurs on the same schedule	A periodic report or a scheduled rule evaluation	Correlate the recurrence against cron and dashboard refresh intervals
Instance restarts repeatedly with no memory alert	OOM kill between reads	Compare `process_start_time_seconds` over time, not RSS

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Ask the second question first

The branch: two numbers, read at the same moment

Branch A - head series climbing with RSS

Branch B - head series flat, RSS climbing

Branch C - it stepped up once and stayed

Branch D - it has been growing for weeks

Choosing the holding action

Pricing the restart

Common patterns

References