Failure: OOM kills, CPU saturation and disk latency

“The server is slow” arrives with no layer attached. Memory, CPU and storage each produce it, they produce it differently, and the wrong diagnosis leads to the wrong purchase order.

Pressure first, then the specific tool

Pressure stall information gives you the contended resource in one file, before you pick a tool:

$ grep . /proc/pressure/*

/proc/pressure/cpu:some avg10=2.11 avg60=1.84 avg300=1.20 total=98213441
/proc/pressure/io:some avg10=61.44 avg60=58.02 avg300=41.77 total=8821334219
/proc/pressure/memory:some avg10=0.00 avg60=0.00 avg300=0.00 total=118442

some is the share of time at least one task was stalled on that resource; full (in the same files) is the share where every task was. In the capture above, 61% of the last ten seconds had something blocked on I/O and nothing was blocked on memory. That is a storage investigation, and load average would not have told you so.

Memory: the OOM kill is written down

An out-of-memory kill is never ambiguous, because the kernel records it:

$ journalctl -k -b | grep -iE 'out of memory|oom-kill|Killed process' | tail

kernel: postgres invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
kernel: Out of memory: Killed process 2841 (java) total-vm:9812344kB, anon-rss:7623112kB, file-rss:0kB, shmem-rss:0kB, UID:998 pgtables:16192kB oom_score_adj:0

Read both lines. The process that invoked the OOM killer is usually the one that asked for memory it could not get. The process that was killed is the one with the highest oom_score - typically the largest resident process, which is often innocent. A database killed because a batch job allocated 20 GB is the classic shape.

Three variants that trip people up:

• Cgroup OOM. With systemd MemoryMax= or a container limit, only that cgroup is out of memory. The host has free memory and free -m looks fine. Look for memory cgroup out of memory in the same log, and check systemctl show UNIT -p MemoryMax -p MemoryPeak.
• No OOM, but heavy reclaim. /proc/pressure/memory is high, vmstat shows si/so swapping, everything is slow and nothing was killed. This is worse for latency than a clean kill.
• A silent death. A process that vanished with no OOM line and no crash was probably killed by something else. Check journalctl -u UNIT for the systemd Result=, and coredumpctl list.

CPU: saturation is a queue, not a percentage

100% CPU utilisation is not a failure. A queue is:

$ vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  0      0 411236  91244 3120448    0    0     8    22 4113 9022 92  6  1  1  0
15  1      0 409884  91244 3120448    0    0     0    16 4088 8974 93  6  0  1  0

r is the number of runnable tasks. Sustained r far above the CPU count means tasks are waiting for a core - real saturation, and latency is already degraded. The same three columns settle the layer:

• High us (user): the application is doing work. Profile it.
• High sy (system): kernel time. Syscall storm, interrupt load, or a pathological workload.
• High wa (iowait): the CPU is idle waiting for storage. This is a storage problem reported by a CPU tool.
• Non-zero st (steal): the hypervisor is not giving you the CPU you asked for. Nothing on the guest can fix it.

Steal time is worth its own habit. On a virtual machine it is the difference between “our application got slower” and “our neighbour got busier”, and only one of those has a fix inside your account.

Disk: saturated is not the same as slow

$ iostat -xz 1 3

Device  r/s    w/s   rkB/s   wkB/s  r_await  w_await  aqu-sz  %util
nvme0n1 12.0  980.0   96.0  125440.0    0.42    18.66   18.30  99.4

• %util near 100 on an SSD or an array means “busy”, not “full”. These devices serve many requests in parallel, so %util saturates long before the device does. It is a misleading number on modern storage.
• await is the latency the application actually experiences. Compare it to what the device should do: sub-millisecond for NVMe, a few milliseconds for SSD, ~10 ms for spinning disk.
• aqu-sz (average queue size) separates the two cases. High await with a deep queue is saturation - you are asking for more than the device can deliver. High await with a shallow queue is a slow device - a failing disk, a degraded array rebuilding, a saturated SAN path, or a throttled cloud volume that has spent its burst credits.

That distinction decides the action. Saturation is fixed by reducing or spreading the load. A slow device is fixed by fixing the device.

1. Read /proc/pressure to name the resource.
2. For memory, confirm or exclude an OOM kill in the kernel log, and check the cgroup limit before the host total.
3. For CPU, read the run queue and split us/sy/wa/st before profiling anything.
4. For storage, read await and queue depth together; treat %util as a hint only.
5. Attribute the load to a cgroup or process before changing capacity - otherwise you buy hardware for a leak.

Knowledge check