Break/Fix 1: Container exits with 137 (OOMKilled)

Reported symptoms

• Container is in Exited (137) state.
• docker inspect shows OOMKilled: true.
• Application logs stop abruptly without an error message.
• Restart policy may show a crash loop.

Evidence provided

$ docker ps -a
CONTAINER ID   STATUS                     NAMES
abc123def456   Exited (137) 2 minutes ago   api

$ docker inspect abc123def456 | grep -i oom
            "OOMKilled": true,

$ docker logs abc123def456 --tail 50
# (no recent log lines — the process was killed before flushing)

Root cause

The container’s cgroup memory limit was exceeded. The kernel’s OOM-killer killed the process. The application did not handle the SIGKILL gracefully, so no cleanup messages were logged.

Resolution

1. Measure actual memory usage.
2. docker stats api --no-stream
3. Compare to the configured mem_limit.
4. Increase the memory limit.
5. Update compose.yml: mem_limit: 1g (was 512m)
6. Or use the Compose shorthand: mem_limit: 1G.
7. docker compose up -d to apply.
8. Investigate the leak.
9. If usage is unbounded, the limit only delays the crash. Profile the application to find the leak.
10. docker exec api ps aux to see RSS per process.
11. docker exec api sh -c "cat /proc/1/status | grep VmRSS" for the container's PID 1.
12. Set appropriate restart policy.
13. restart: on-failure:5 (5 attempts, then stop). Prevents crash loops from masking other issues.

Verification

1. Container stays running under normal load. docker ps shows Up X minutes.
2. Memory usage stays below the limit. docker stats shows usage below the limit with headroom.
3. The leak (if any) is fixed. Memory usage is stable over time.
4. The OOMKilled flag is false. docker inspect does not show OOMKilled: true in subsequent runs.

Prevention

• Set memory limits based on actual usage + headroom, not guesswork.
• Monitor memory usage; alert at 80% of the limit.
• Investigate growth; memory usage that increases over time is a leak.
• Use memory reservations (mem_reservation) for soft limits.