Break/Fix: the API returns 502 a few times an hour and nothing is in the application log

Reported symptoms

You are handed a ticket, not a diagnosis:

• The API returns 502 for a handful of requests, a few times an hour, then recovers on its own.
• No pattern by endpoint, client or payload size that anyone has been able to find.
• The service is running. systemctl status api says active (running) and shows no restarts.
• The application log has nothing: no error, no stack trace, no shutdown message around any of the 502s.
• The load balancer health check passes throughout.
• free -m shows several GB available on a 31 GB host.

The developer’s position is that the platform is dropping connections. The platform team’s position is that the application is crashing. Both are looking at evidence that supports them.

Evidence provided

$ systemctl status api
● api.service - Example API
     Loaded: loaded (/etc/systemd/system/api.service; enabled)
     Active: active (running) since Sat 2026-08-02 09:14:31 UTC; 9 days ago
   Main PID: 1187 (api-server)
      Tasks: 11 (limit: 38314)
     Memory: 1.9G

$ journalctl -u api --since "-2h" | grep -iE 'error|fatal|panic|exit'
(no output)

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           31842       18204        1120         312       12518       12034

$ journalctl -k --since "-2h" | grep -i oom
Aug 11 09:07:44 host kernel: api-worker invoked oom-killer: gfp_mask=0x1100cca, order=0
Aug 11 09:07:44 host kernel: memory: usage 2097152kB, limit 2097152kB, failcnt 4193
Aug 11 09:07:44 host kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=api.service,mems_allowed=0,oom_memcg=/system.slice/api.service,task_memcg=/system.slice/api.service,task=api-worker,pid=20714,uid=997
Aug 11 09:41:02 host kernel: oom-kill:constraint=CONSTRAINT_MEMCG,...,task=api-worker,pid=21590,uid=997

$ cat /sys/fs/cgroup/system.slice/api.service/memory.events
low 0
high 0
max 4193
oom 51
oom_kill 47

$ cat /sys/fs/cgroup/system.slice/api.service/memory.max
2147483648

$ cat /sys/fs/cgroup/system.slice/api.service/memory.peak
2147483648

$ systemctl show api -p MemoryMax -p MemoryHigh -p MemoryAccounting -p OOMPolicy -p Restart
MemoryMax=2147483648
MemoryHigh=infinity
MemoryAccounting=yes
OOMPolicy=continue
Restart=no

Root cause

Three settings combine into a fault that is invisible from every place an operator normally looks.

The limit is the cgroup’s, not the host’s. The kernel line says constraint=CONSTRAINT_MEMCG and names oom_memcg=/system.slice/api.service. This is not a machine-wide out-of-memory event; it is the unit hitting its own MemoryMax=2G. That is why free -m looks healthy and why host-level memory alerts never fired. memory.max in the cgroup is 2147483648 — 2 GiB — and memory.peak sits exactly on it, which is the signature of a workload pressed against a ceiling rather than one that comfortably fits.

A worker died, not the service. The kernel picked task=api-worker inside the cgroup, not the main PID. The unit therefore never entered a failed state. OOMPolicy=continue tells systemd to take no action when a process in the unit is OOM-killed, so systemd did not stop or restart anything either. The default is stop; somebody changed it, probably to stop the service flapping — which worked, in the sense that the flapping became invisible.

SIGKILL leaves no note. An OOM kill is SIGKILL. The process gets no chance to flush a log line, run an exception handler, or drain its connections. The requests it was serving die mid-flight and the load balancer records 502. The application log is empty because there was never a moment in which the application could write to it.

memory.events tells the longer story. max 4193 counts the times allocation hit the limit; oom_kill 47 counts the times a process was actually killed. The workload has been pressed against the ceiling thousands of times, reclaiming hard each time, long before it started losing workers. The 502s are the late symptom of a limit that stopped fitting weeks ago.

Resolution

1. Measure before you choose a number. cat /sys/fs/cgroup/system.slice/api.service/memory.peak gives the high-water mark since the last reset, but it is pinned at the limit here so it only tells you the workload wants at least 2 GiB. Take the real figure from your metrics history at peak traffic, or lift the limit temporarily and observe. Guessing a new limit reproduces the same incident at a different number
2. Set the two limits, not one. In a drop-in at /etc/systemd/system/api.service.d/10-memory.conf:
3. ``ini [Service] MemoryAccounting=yes MemoryHigh=5G MemoryMax=6G OOMPolicy=stop Restart=on-failure RestartSec=5s ``
4. Understand why both. MemoryHigh is a throttle: crossing it puts the cgroup under heavy reclaim pressure and slows it down, which is visible and recoverable. MemoryMax is a wall: crossing it kills. With MemoryHigh below MemoryMax you get a degraded, observable state before you get a dead process
5. Restore the default OOM behaviour. OOMPolicy=stop makes an OOM kill stop the unit; Restart=on-failure brings it back and records the failure. The incident becomes loud, which is the point
6. Apply it. sudo systemctl daemon-reload && sudo systemctl restart api
7. Add the alert that was missing. Export the cgroup oom_kill counter and alert on any increase - see below

$ systemctl show api -p MemoryHigh -p MemoryMax -p OOMPolicy -p Restart

MemoryHigh=5368709120
MemoryMax=6442450944
OOMPolicy=stop
Restart=on-failure

Export the counter so the next occurrence cannot hide. A textfile-collector script on a timer is enough:

#!/usr/bin/env bash
# /usr/local/sbin/cgroup-oom-metrics
set -euo pipefail
out=$(mktemp)
for f in /sys/fs/cgroup/system.slice/*.service/memory.events; do
  unit=$(basename "$(dirname "$f")")
  kills=$(awk '$1=="oom_kill"{print $2}' "$f")
  printf 'cgroup_memory_oom_kill_total{unit="%s"} %s\n' "$unit" "$kills"
done > "$out"
mv "$out" /var/lib/node_exporter/textfile/cgroup_oom.prom

- alert: CgroupOOMKill
  expr: increase(cgroup_memory_oom_kill_total[15m]) > 0
  for: 0m
  labels: { severity: critical }
  annotations:
    summary: "{{ $labels.unit }} on {{ $labels.instance }} lost a process to the cgroup OOM killer"

Verification

1. Confirm the limits are live. systemctl show api -p MemoryHigh -p MemoryMax -p OOMPolicy returns the new values, not the old ones
2. Watch the counter, not the dashboard. Record oom_kill from memory.events now, then again after a full peak-traffic period. It must not have moved
3. Confirm the symptom is gone at the edge. The 502 rate at the load balancer returns to zero over the same window. A quiet cgroup with continuing 502s means there is a second fault
4. Confirm reclaim pressure is not the new steady state. memory.events high should be low; a large and growing high count means the service is now permanently throttled and needs a real capacity increase rather than a bigger ceiling
5. Prove the detection works. In staging, run the service against a deliberately small MemoryMax until it is killed. The unit must enter failed, restart, and the alert must fire. An untested alert is not a control

Prevention

• Every MemoryMax traces back to a measurement. A number someone picked because it looked round will be wrong, and the only question is when.
• Set MemoryHigh below MemoryMax so throttling precedes killing. A slow service is an incident you can catch; a killed worker is one you find out about from users.
• Leave OOMPolicy at the default stop. If you find continue in a unit file, treat it as a finding: someone hid a failure rather than fixing it.
• Alert on memory.events oom_kill for every unit that has a memory limit. Unit state is not sufficient — this whole scenario happened with the unit reporting active throughout.
• Re-review cgroup limits whenever a service’s workload profile changes. Limits are sized against a workload, and the workload moves.