CPU and memory performance engineering · Proxmox VE

Why this matters in production

“The VM is slow” tickets are the most common in production. Most are diagnosed by measuring CPU, memory, storage, or network — and fixing the wrong one wastes hours.

CPU and memory are where the wrong fix is most tempting, because both have obvious-looking levers — pin the vCPUs, add hugepages, raise the memory — that feel like progress and frequently make things worse.

CPU contention

Inside the guest, look for:

%steal: time the vCPU wanted to run but the host did not schedule it.
%user vs %system: user = application; system = kernel. Heavy system time usually means I/O or memory issues.

On the host, look for:

top / htop: per-CPU utilisation. If all CPUs are 100 %, the host is saturated.
mpstat: per-CPU breakdown. Useful for detecting single-core hotspots.
pidstat: per-process CPU. Useful for finding noisy neighbours.

Read-only / Safeper-CPU breakdown and whether anything is waiting— Read-only. mpstat -P ALL exposes single-core hotspots that an aggregate figure hides. The pressure file answers a different question — whether tasks are stalled — which utilisation cannot.

mpstat -P ALL 1 5

cat /proc/pressure/cpu
uptime
nproc

High steal does not always mean the host is oversubscribed

%steal is time a vCPU was runnable and not scheduled. Two very different causes produce it, and they need opposite responses.

The host is genuinely contended. Host CPU utilisation is high, /proc/pressure/cpu shows non-zero some avg10, and other guests on the node are also complaining. This is the case everyone assumes, and the fix is capacity or contention management.

A cpulimit is throttling this guest. The host is idle, pressure is zero, and no other guest is affected. A cgroup CPU quota stops every vCPU in the guest when the quota is exhausted, and from inside the guest that is indistinguishable from host contention — the guest kernel has no way to observe why it was descheduled.

The discriminator takes one command: read nr_throttled in the guest’s cgroup cpu.stat. If it is climbing while the host is idle, a cap set during some earlier incident is still in force. xvii-performance-noisy-neighbours covers it.

Treating the second case as the first leads to buying capacity for a node that has plenty.

Memory pressure

Inside the guest:

free -h: total and available memory.
vmstat 1: si/so (swap in/out) columns; non-zero = pressure.
/proc/pressure/memory: kernel-side stall metrics.

On the host:

Per-VM RSS via ps or top.
Host free memory (free -h).
Host swap usage (should be near zero).

flowchart LR
  A[Guest VM] -->|balloon| B[Host memory]
  B --> C[Physical RAM?]
  C -->|no| D[Swap / OOM]

NUMA effects

On multi-socket hosts, a vCPU running on socket 0 accessing memory attached to socket 1 incurs a penalty — typically 1.5 to 2 times the local latency.

Read-only / Safetopology, and whether allocations are landing off-node— Read-only. numa_miss and numa_foreign climbing steadily means memory is being allocated on the wrong node for the task using it. numastat -c qemu attributes it per QEMU process.

numactl --hardware

numastat
numastat -c qemu

For latency-sensitive VMs:

Pin vCPUs to specific cores.
Enable NUMA awareness in the VM config (qm set VMID --numa 1).
Ensure VMs and their memory fit in one NUMA node.

A VM larger than one NUMA node cannot avoid remote access; the best you can do is give it a matching virtual topology with --numa 1 so the guest’s own scheduler knows about the split and can place its processes accordingly.

CPU pinning

Configuration changepin a guest's vCPUs to specific host cores— Applies live. The value is a taskset-style host CPU list; the documentation gives 0,5,8-11 as an example. Note the Proxmox documentation's warning that CPU affinity is not a security feature — it constrains scheduling, not access.

VMID=100

qm set "$VMID" --affinity 0-3

qm config "$VMID" | grep -E 'affinity|numa|cores'

Pinning reduces context switches and improves cache locality. Trade-off: less flexibility for the scheduler.

Hugepages

Hugepages reduce TLB pressure for large-memory workloads. There are two different mechanisms with the same name, and conflating them is the usual source of error.

	Transparent hugepages (THP)	Explicit hugepages
Configured	Host-wide, `/sys/kernel/mm/transparent_hugepage/`	Per VM, `qm set --hugepages`
Allocation	Opportunistic, by `khugepaged`	Reserved from a pool
Guest memory	Still balloonable, still KSM-mergeable	Pinned; ballooning disabled
Cost	Effectively free	Memory reserved whether used or not

THP is on by default and is generally a win on a KVM host; xvii-performance-host-tuning covers the defrag setting that decides whether it also causes latency spikes.

Explicit hugepages are a deliberate per-guest choice.

Service impact possiblegive a VM explicit hugepages— Takes effect at the next guest start, not live. The documented values are 2 (2 MiB), 1024 (1 GiB) or any. There is no --hugepages 1: the number is the page size in MiB, not a boolean. This pins the guest's memory and disables ballooning for it.

VMID=100

# 2 MiB pages
qm set "$VMID" --hugepages 2

# or 1 GiB pages, if the host has them reserved
qm set "$VMID" --hugepages 1024

qm config "$VMID" | grep -E 'hugepages|balloon'

1 GiB hugepages must be reserved at boot, or the VM will not start

2 MiB hugepages can usually be allocated on demand. 1 GiB hugepages generally cannot: they require a gigabyte of physically contiguous memory, and a host that has been running VMs for weeks does not have that. The allocation fails, and the symptom is a VM that refuses to start with a memory allocation error — on a host with plenty of free memory.

They have to be reserved on the kernel command line at boot, before memory fragments:

default_hugepagesz=1G hugepagesz=1G hugepages=32

That reserves 32 GiB, permanently, unavailable to anything that is not using 1 GiB hugepages — including every other guest on the node and the host itself. It is a real capacity decision, not a tuning knob, and it belongs in the node’s build documentation.

Verify what is actually reserved before relying on it:

grep -i huge /proc/meminfo
ls /sys/kernel/mm/hugepages/

Read-only / Safewhat hugepages exist, at what size, and how many are free?— Read-only. Hugepagesize is the default size on this host. The per-size directories under /sys show nr_hugepages (reserved) and free_hugepages (available) for each supported size independently.

grep -iE 'huge|AnonHugePages' /proc/meminfo

for D in /sys/kernel/mm/hugepages/*/; do
echo "== $D"
cat "$D/nr_hugepages" "$D/free_hugepages"
done

Read-only / Safea host with no explicit hugepages, but THP in heavy use— Illustrative. HugePages_Total 0 means no explicit pool is reserved. AnonHugePages of 184 GiB means transparent hugepages are backing most of the guest memory, which is the desirable state on a KVM host.

# grep -iE 'huge|AnonHugePages' /proc/meminfo

AnonHugePages:  193273528 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

Illustrative output

Overcommit tuning

A host with 256 GB RAM running 12 VMs configured with 32 GB each is committing 384 GB to 256 GB of physical. This works if ballooning is effective and guests are not all saturated simultaneously.

Measure first:

Read-only / Safewhat is committed against what exists— Read-only. Committed_AS is what the kernel has promised; MemTotal is what exists. On a hypervisor the more useful figure is the sum of guest maximum memory, since a guest can claim its maximum at any time.

grep -E 'MemTotal|MemAvailable|CommitLimit|Committed_AS' /proc/meminfo

pvesh get /cluster/resources --type vm --output-format json | jq '
[ .[] | select(.status=="running") ] | { guests: length, maxmem_gib: ( map(.maxmem) | add / 1073741824 | floor ) }'

Production considerations

Under the hood

Why pinning a VM’s vCPUs helps less than expected, and sometimes hurts.

A VM’s vCPUs are threads in a QEMU process, and the Linux scheduler treats them like any other threads. --affinity restricts which host CPUs those threads may run on, using the same mechanism as taskset.

Two effects are commonly assumed and only one of them is real.

Cache locality is real. A vCPU thread that keeps returning to the same physical core finds its working set still in that core’s L1 and L2 caches. Migrating between cores means a cold cache on arrival, and migrating between sockets means a cold last-level cache plus, potentially, remote memory access. For a latency-sensitive guest this is measurable.

“Guaranteed CPU” is not. Pinning does not reserve anything. Pinning VM 100 to cores 0-3 does not stop VM 101, the host kernel, a Ceph OSD or ksmd from also running on cores 0-3. It only constrains where VM 100 may go. Pinning without also constraining everything else away from those cores gives you the restriction and not the isolation — and the restriction alone makes the guest worse off, because it can no longer escape to an idle core when its assigned ones are busy. The Proxmox documentation notes separately that CPU affinity is not a security feature.

Genuine isolation requires keeping other work off those cores as well: isolcpus on the kernel command line, or explicitly pinning the other guests elsewhere, plus attention to where the host’s own threads land. That is a whole-node configuration rather than a per-guest setting, which is why pinning belongs on nodes dedicated to latency-sensitive workloads rather than being sprinkled across a general-purpose cluster.

On the memory side, the reason hugepages help a guest at all is the two-level address translation. A guest virtual address becomes a guest physical address through the guest’s page tables, and that becomes a host physical address through the EPT or NPT. A full walk with 4 KiB pages at both levels is up to 24 memory references for a single uncached translation. Backing guest memory with 2 MiB host pages removes a level from the host-side walk and reduces the TLB entries needed to map the same guest RAM by a factor of 512.

That also explains why the benefit is concentrated in guests with large, randomly accessed working sets — big databases, in-memory caches — and close to invisible for a web server whose hot set fits in the TLB either way. It is worth measuring the guest before reserving gigabytes of host memory for it.

What happens if you pin a latency-sensitive VM and it later fails over?

The pinning follows the guest to a node where it may mean something completely different, and nothing warns you.

The configuration is portable; the topology is not. affinity: 0-3 is a list of host CPU numbers. On the original node those might be four cores on socket 0, adjacent, sharing an L3 cache, with the guest’s memory local to them. On the failover target, cores 0-3 might be spread across two sockets depending on how the BIOS enumerates them, or might be the cores where the Ceph OSDs were deliberately placed, or — on a node with a smaller CPU — might be a quarter of the entire machine.

The failure modes, in increasing order of subtlety. If the target node has fewer CPUs than the highest number in the affinity list, the guest may fail to start at all, which is at least loud. More often it starts and runs badly: pinned onto cores already busy with host work, unable to migrate away, showing high steal time on a node that looks idle. And in the worst case it runs on cores whose socket does not hold its memory, paying remote access on every miss, at perhaps 60 % of the performance it had — with no error anywhere and a configuration that is byte-for-byte identical.

Why the diagnosis is hard. Nothing changed. The VM configuration is the same, the application is the same, and the only difference is a node name that nobody thinks to correlate. The ticket says “it got slow last Tuesday” and Tuesday is when HA moved it.

What to do about it. Three options, in decreasing order of robustness.

Constrain where the guest can go: an HA group restricted to nodes with identical topology means the affinity list always means the same thing. This is the correct answer and it is usually available, because latency-sensitive guests generally should not be roaming anyway.

Verify topology uniformity across the nodes the guest can reach — lscpu -e and numactl --hardware on each — and keep it uniform as hardware is refreshed. This is maintenance work that is easy to forget across a hardware generation.

Or drop the pinning and use --numa 1 alone, which gives the guest a virtual topology matching its size without hard-coding host CPU numbers. This loses some cache locality and gains portability, and for many guests it is the better trade.

The general principle. Any setting that references host-specific identifiers — CPU numbers, PCI addresses, device paths, bridge names — is a hidden coupling between a guest and a node. Live migration and HA both assume guests are portable, and every such setting is an exception to that assumption which should be either removed or explicitly fenced with an HA group.

Common mistakes

Reading %steal as proof the host is oversubscribed. A cpulimit produces identical symptoms on an idle host; check nr_throttled.
Tuning without measuring. On a hypervisor an unmeasured change helps one guest at every other guest’s expense.
Pinning everything. Pinning restricts, it does not reserve; without also keeping other work off those cores it makes the guest worse.
Using --hugepages 1. The value is a page size in MiB — 2, 1024 or any.
Expecting 1 GiB hugepages to be allocated on demand. They need reserving at boot on the kernel command line, and the reservation is unavailable to everything else.
Forgetting hugepages disable ballooning and pin the guest’s memory.
Confusing THP with explicit hugepages. One is host-wide and free; the other is per-guest and costs reserved memory.
Believing free -h on a ZFS node. The ARC is not reclaimable at page-cache speed.
Counting ballooning and KSM savings as capacity. Both stop working during a failover.

Key takeaways

Measure first: mpstat -P ALL, vmstat, numactl --hardware, and /proc/pressure/cpu for whether anything is actually waiting.
High %steal means the vCPU was not scheduled — by a busy host, or by a cpulimit on an idle one. nr_throttled distinguishes them.
Memory pressure shows as swap activity; on a ZFS node also read arcstat size against MemAvailable.
NUMA locality matters on multi-socket hosts; a guest larger than one node cannot avoid remote access, so give it --numa 1 and let the guest schedule around it.
Pinning constrains rather than reserves. Pin only latency-sensitive guests, on nodes where you also control what else runs there.
--hugepages takes 2, 1024 or any — a page size in MiB. 1 GiB pages must be reserved at boot and are unavailable to anything else.
Explicit hugepages pin guest memory and disable ballooning; THP does neither.
Overcommit that works in steady state fails during a failover, because ballooning and KSM both stop helping at the same moment.