Linux memory pressure and the OOM killer · PostgreSQL

PostgreSQL asks the operating system for memory and assumes it gets what it is given. Linux, by default, promises more than it has. The gap between those two positions is where clusters die.

Reading the limit that actually applies

Read-only / Safewhat a containerised PostgreSQL can see— Captured on 18.6 inside the test container. The container has no memory limit set; the host has 47 GB.

$ cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current; free -m | head -2

  /sys/fs/cgroup/memory.max      = max
/sys/fs/cgroup/memory.high     = max
/sys/fs/cgroup/memory.current  = 2070110208
/sys/fs/cgroup/memory.swap.max = max

             total        used        free      shared  buff/cache   available
Mem:           47145        9397        7769         416       30979       37747

free reports the host’s 47 GB. It does not know about cgroups, and it never has. A capacity calculation performed inside a container using free will size shared_buffers and work_mem for memory the process may not be allowed to use.

The authoritative figures are in the cgroup filesystem:

# cgroup v2, which is what modern distributions use
cat /sys/fs/cgroup/memory.max        # hard limit; "max" means unlimited
cat /sys/fs/cgroup/memory.high       # throttling threshold
cat /sys/fs/cgroup/memory.current    # current usage
cat /sys/fs/cgroup/memory.swap.max   # swap allowance

# cgroup v1, still present on older systems
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

On this container memory.max is max — unlimited — which is exactly why free’s answer looks plausible and is still the wrong place to read it from. On a Kubernetes pod with a memory limit, the two numbers differ by an order of magnitude and nothing warns you.

memory.high and memory.max fail differently

memory.max is a hard limit. Exceeding it triggers the cgroup OOM killer, which selects a process inside the cgroup and kills it.

memory.high is a throttling threshold. Exceeding it does not kill anything; the kernel puts the cgroup under heavy reclaim pressure and stalls its processes. The symptom is a database that becomes extremely slow with no lock contention, no I/O wait attributable to storage, and no obvious cause — because the stall is in the kernel’s memory reclaim path.

A Kubernetes memory request maps to a soft guarantee and a memory limit maps to memory.max. A pod whose usage sits between them is not killed and is not healthy either.

If a containerised PostgreSQL is inexplicably slow, read memory.current against memory.high before anything else.

Overcommit

Read-only / Safethe overcommit settings on the test host— Captured on 18.6. These are the Linux defaults.

$ cat /proc/sys/vm/overcommit_memory /proc/sys/vm/overcommit_ratio

0
50

`vm.overcommit_memory`	Behaviour
`0` (default)	Heuristic. Allows most allocations; kills later if the promise cannot be kept
`1`	Always allow. Never refuse an allocation
`2`	Never overcommit. Refuse allocations beyond swap + `overcommit_ratio`% of RAM

The PostgreSQL documentation recommends 2 for dedicated database hosts, and the reasoning is worth understanding rather than following.

Under 0, an allocation that cannot ultimately be honoured succeeds anyway, and the process is killed later when it touches the memory. The failure arrives at an arbitrary moment, in an arbitrary process, with no relationship to which process asked for too much.

Under 2, the allocation fails at the point of asking. PostgreSQL receives an out-of-memory error, the query aborts with a clear message, and everything else keeps running.

sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=80    # with swap, tune to the host

The caution: setting 2 with overcommit_ratio too low makes allocations fail that would have been fine, and PostgreSQL will not start if it cannot get its shared memory. Compute the allowance deliberately: swap + (RAM × overcommit_ratio / 100).

What the OOM killer does to PostgreSQL

When the kernel kills a process to reclaim memory it chooses by a score derived from memory usage, adjusted by oom_score_adj.

If it selects a backend, the consequence is out of proportion to the loss of one connection:

LOG:  server process (PID 12345) was terminated by signal 9: Killed
DETAIL:  Failed process was running: SELECT ...
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing

The postmaster restarts every backend. A backend killed without cleanup may have left shared memory inconsistent, and the postmaster cannot prove otherwise, so it recycles the whole cluster. One killed query becomes a full disconnection event for every session.

This is the same mechanism as kill -9 from lesson IX-06, and it is why that instruction exists.

If it selects the postmaster, the cluster is simply gone.

Protect the postmaster, not the backends

Linux allows a per-process adjustment, from −1000 (never kill) to +1000 (kill first).

# in the systemd unit for postgresql
[Service]
OOMScoreAdjust=-900

PostgreSQL then re-raises the score for its child backends, using PG_OOM_ADJUST_FILE and PG_OOM_ADJUST_VALUE, so that a backend is chosen before the postmaster:

Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0

The reasoning: if something must die, losing one backend and a cluster restart is bad; losing the postmaster is worse, and on a cluster with no supervision it is an outage until someone notices.

On the test container the postmaster’s oom_score_adj is 0 — the default, and unprotected. Check yours:

cat /proc/$(pgrep -o postgres)/oom_score_adj

Recognising it afterwards

An OOM kill leaves evidence in three places, and the PostgreSQL log alone is ambiguous.

The kernel log is definitive:

dmesg -T | grep -i -E 'killed process|out of memory|oom'
journalctl -k --since '1 hour ago' | grep -i oom

The PostgreSQL log shows the consequence:

LOG:  server process (PID 12345) was terminated by signal 9: Killed

signal 9 with no operator action is the signature. A pg_terminate_backend produces signal 15 and a clean FATAL message instead.

The cgroup counters, on a containerised deployment:

grep oom /sys/fs/cgroup/memory.events
# oom 3
# oom_kill 3

Non-zero and increasing means the container has been hitting its limit, whether or not anyone noticed.

Swap: not zero, and not large

The instinct on a database host is to disable swap entirely, so that nothing important is ever paged out. It is the wrong instinct in both directions.

With no swap, the kernel has no way to reclaim anonymous pages under pressure, so it goes directly from “tight” to OOM kill. There is no degraded mode.

With large swap, a database that starts paging becomes so slow that it is effectively down, while appearing up — which is worse than a clean failure because it does not trigger the failover that a dead cluster would.

A small swap allocation gives the kernel somewhere to put genuinely cold pages and a little time before it must kill something. Combine it with:

sysctl -w vm.swappiness=1     # use swap only under real pressure

vm.swappiness=1 rather than 0: zero disables swap for anonymous pages almost entirely on modern kernels, which reintroduces the first problem.

Huge pages, from lesson XI-01, are not swappable at all, which is a further argument for using them: the buffer pool cannot be paged out no matter what the kernel decides.

Why the postmaster restarts everything, and what survives

The postmaster distinguishes two kinds of child exit.

A backend exiting normally, or with a clean FATAL, has released its locks, cleaned up its shared memory references and left the buffer pool consistent. The postmaster reaps it and carries on.

A backend killed by a signal has done none of that. It may have been holding a buffer pinned, midway through a page write, or holding an LWLock protecting a shared structure. The postmaster cannot inspect shared memory to find out, so it assumes the worst: it terminates every other backend, discards and reinitialises shared memory, and runs recovery.

What this costs and what survives:

Every connection is dropped. Applications see a connection reset, and pooled connections are all invalidated at once, which frequently produces a thundering herd of reconnections at exactly the wrong moment.

Committed data is safe. Recovery replays WAL from the last checkpoint, so nothing committed is lost. This is a crash, and crash recovery is the mechanism Part XII covers.

Uncommitted transactions are lost, which is correct.

The recovery duration depends on checkpoint distance, so a cluster tuned for infrequent checkpoints has a longer restart. That trade is made in Part XII, and this is one of the cases where it is paid.

The operational conclusion: an OOM kill of one backend is not a small event, and treating it as “one query died” underestimates it by the number of connections on the cluster. That is the argument for sizing work_mem conservatively per role rather than generously cluster-wide — the failure is not proportionate to the mistake.

What to take from this

free inside a container reports the host. Read memory.max from the cgroup filesystem.
memory.high throttles rather than kills, and presents as inexplicable slowness.
vm.overcommit_memory=2 turns a later arbitrary kill into an immediate honest allocation failure.
An OOM kill of one backend restarts every backend on the cluster.
Protect the postmaster with OOMScoreAdjust, and let PostgreSQL re-raise the score for its children.
Small swap with vm.swappiness=1. Not zero, not large.
dmesg is definitive; signal 9 in the PostgreSQL log is the signature.

Cross-course references

Linux for Production Sysadmins — Part XL (Memory Performance) covers overcommit, the OOM score, and reading the kernel’s own record of the kill, which is the primary evidence for this failure.
Docker & Containers — Part XV (Resource controls) covers the cgroup limit that triggers the kill inside a container, where the host has memory to spare and the container does not.
Kubernetes for Production Sysadmins — Part XXXII (Node pressure and eviction) and Part XIII (QoS classes) cover which pod is chosen first, which is a decision made outside PostgreSQL entirely.