Skip to main content
RunBook Academy

← All runbooks in Proxmox VE

medium riskservice affecting~30 min

Diagnose and fix memory pressure on a PVE host

1 · Prerequisites

Confirm every item is in place before any state change.

  • A PVE host reporting high memory usage or OOM killer activity
  • Root SSH access to the host

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · free -h shows high used memory
  • · dmesg | grep -i oom shows recent kills
  • · cat /proc/pressure/memory shows high some pressure
  • · Identify which VMs are ballooning: pvesh get /nodes/<host>/qemu/*/status/current | jq .balloon

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the heaviest VM: pvesh get /cluster/resources --type vm --output-format json | jq -r ".[] | select(.status==\"running\") | [.name, .mem, .maxmem] | @tsv" | sort -k2 -n -r | head -5
  2. 2Check for memory leaks in guest apps: top inside the VM, look for processes with growing RSS
  3. 3If a specific VM is the issue, reduce its allocation: qm set <vmid> --memory <smaller value>
  4. 4If all VMs are healthy but the host is still pressured, check transparent hugepage settings and NUMA balancing
  5. 5For sustained pressure, add physical RAM or move workloads to other hosts
  6. 6For temporary spikes, increase swap or set overcommit on a per-VM basis

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓free -h shows available memory > 20% of total
  • ✓dmesg shows no recent OOM events
  • ✓Memory pressure in /proc/pressure/memory is in the "some" category below 20%
  • ✓VM workloads run normally with no degradation

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If reducing VM memory breaks an application, restore the previous value and investigate further
  • ↶For NUMA changes, verify the application is single-threaded before disabling NUMA balancing

6 · Escalation

When the runbook isn't enough, contact:

  • · Application owner for in-guest memory leaks
  • · Capacity planning if sustained pressure requires more hardware

Diagnose and fix memory pressure

Memory pressure manifests as either slow performance (steal time) or hard OOM kills. The diagnostic path differs for each.

Soft pressure (slow performance)

  • Check iostat for high await on the storage backend
  • Check VM-side memory metrics for ballooning
  • Consider transparent hugepage and kernel same-page merging settings

Hard pressure (OOM kills)

  • dmesg | grep -i oom shows the killed processes
  • Check journalctl -k for the same period
  • The kernel OOM killer favours killing the process with the highest oom_score, which is often the database or the biggest VM

Mitigation order

  1. Identify the offending VM (if any)
  2. Reduce its memory allocation if possible
  3. Investigate in-guest memory leaks
  4. Move workloads to other hosts
  5. Add physical RAM (last resort)

References

  1. Memory