Host tuning: governors, C-states, THP and NUMA balancing · Proxmox VE

There is a class of performance ticket that survives every VM-level fix. The guest has enough vCPU. It has enough RAM. Its disk is on NVMe. The host graph is flat at 30% CPU. And the application still misses its latency target by a factor of three.

The settings in this lesson are the usual answer. They live on the host, they apply to every guest on it, and none of them show up in qm config. An operator who has only ever tuned VMs cannot find them, which is why this ticket gets escalated rather than solved.

The one idea that makes the rest make sense

The Linux course states it as a rule and it is worth repeating here because every measurement in this lesson depends on it:

Utilisation is not saturation.

A host at 30% CPU utilisation is not a host with 70% of its performance in reserve. Utilisation says what fraction of wall-clock time a CPU spent executing something. It says nothing about how fast it executed, nor about how long a task waited before it got a turn. A node parked at its lowest frequency with deep idle states enabled is at 30% utilisation and delivering perhaps a third of the work per second it is capable of.

The four levers below each attack a different version of that gap.

Lever	The gap it closes	Costs
Frequency governor	Clock speed under bursty load	Power, heat
C-states / idle policy	Wakeup latency, measured in microseconds	Power, heat
Transparent hugepages	Guest memory address-translation overhead	Latency spikes if `defrag` is wrong
NUMA balancing	Remote memory access on multi-socket hosts	Page-fault overhead; fights explicit pinning

Measure first: what is this node actually doing?

Read-only / Safethe four numbers, before you change anything— Read-only. Governor and driver come from sysfs; the achieved frequency is what the CPU is really running at, which is the only one of these that is an outcome rather than a setting.

# Which scaling driver and governor?
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# What is every core actually clocked at, right now?
grep 'MHz' /proc/cpuinfo | sort -u -t: -k2 -n | head

# Which idle states exist, and how deep do they go?
cpupower idle-info

# The consolidated view
cpupower frequency-info

Read-only / Safea node that is quietly capped— Illustrative. Note the driver: with intel_pstate in active mode the governor names are the pstate driver's own, not the generic cpufreq ones, and 'powersave' here does not mean what it means under acpi-cpufreq.

# cpupower frequency-info

analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency:  Cannot determine or is not supported.
hardware limits: 800 MHz - 3.70 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 3.70 GHz.
                The governor "powersave" may decide which speed to use
                within this range.
current CPU frequency: 1.19 GHz (asserted by call to hardware)
boost state support:
  Supported: yes
  Active: yes

Illustrative output

The line that matters is the last frequency reading, not the governor name. A busy node under the powersave governor should still be near its maximum; a node sitting at 1.19 GHz while the application complains is either genuinely idle between bursts, or the frequency ramp is losing the race against a bursty workload.

Saturation, not utilisation: read PSI

Pressure Stall Information is the metric that answers “is anything waiting?” — which is the question utilisation cannot answer.

Read-only / Safeis anything actually stalled?— Read-only. The 'some' line is the share of time at least one task was stalled on that resource; 'full' is the share where every runnable task was stalled. Non-zero avg10 on cpu with low utilisation is the signature this lesson is about.

cat /proc/pressure/cpu
cat /proc/pressure/io
cat /proc/pressure/memory

Read-only / Safe30% utilisation, and tasks are still waiting— Illustrative. 8% of time with something stalled on CPU, on a host reporting 30% utilisation, means the CPUs are not delivering what the utilisation figure implies.

# cat /proc/pressure/cpu

some avg10=8.11 avg60=7.44 avg300=6.90 total=48213991

Illustrative output

Lever 1 — the frequency governor

On modern Intel and AMD server parts the relevant driver is intel_pstate or amd-pstate, and it exposes only two governor names: powersave and performance. Both are hardware-driven; powersave here is not the classic cpufreq powersave that pins the CPU to its minimum. It is a demand-following policy, and it is the right default for most hosts.

performance pins the requested P-state to the maximum. It does not change your electricity bill by a rounding error — on a dense node it is tens of watts per socket, continuously.

Configuration changeset the governor on every core— Changes CPU frequency policy immediately for all cores. Reversible by setting it back. This does not survive a reboot on its own; the persistence section below covers that.

cpupower frequency-set -g performance

# verify it took, on every CPU rather than just cpu0
grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | sort -u

Lever 2 — C-states and wakeup latency

This is the lever most often needed and least often understood, because its symptom does not look like a performance problem at all.

A CPU in a deep idle state (C6, C7) has powered down parts of its cache and clock domains. Waking it takes time — tens to low hundreds of microseconds. That is invisible to a throughput benchmark, which keeps the core busy so it never idles. It is very visible to a workload made of many short, latency-sensitive wakeups: small network requests, a synchronous database commit, an NFS or Ceph client round trip.

Read-only / Safewhich idle states, and how long does exit take?— Read-only. The 'Latency' figure per state is the documented exit latency in microseconds. Compare it against your application's latency budget before deciding it does not matter.

cpupower idle-info

# how much time is actually spent in each state?
grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/name
grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/latency
grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/time

Read-only / Safeexit latency is not a rounding error— Illustrative. C6 costs 133 microseconds to leave. An application with a 1 ms budget that wakes four times per request has just spent half of it waiting for cores to come back.

# cpupower idle-info

CPUidle driver: intel_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 4
Available idle states: POLL C1 C1E C6
POLL:
Latency: 0
C1:
Latency: 2
C1E:
Latency: 10
C6:
Latency: 133

Illustrative output

There are two places to limit idle depth, and they are not equivalent.

In firmware. The BIOS setting (often “Package C-State Limit”, “CPU Power Management” or a “Performance” system profile) is the durable answer and the one that also governs package-level states the OS cannot reach. Part II covers it as a pre-install decision, and it is the right place to make it.

On the kernel command line. intel_idle.max_cstate=1 or the generic processor.max_cstate=1 caps how deep the OS will go. This is the reversible one, which makes it the right tool for testing whether C-states are implicated before you book a firmware window.

Service impact possiblecap idle depth for a test — GRUB installs— Requires a reboot to take effect, so the node must be drained first. Reversible by removing the parameter and re-running update-grub. Check which bootloader this node uses before editing: proxmox-boot-tool status will tell you.

proxmox-boot-tool status

# GRUB installs only — edit GRUB_CMDLINE_LINUX_DEFAULT, then:
update-grub

Service impact possiblecap idle depth for a test — systemd-boot installs— Requires a reboot. /etc/kernel/cmdline is a single line; append to it rather than replacing it, and refresh so every ESP is updated.

cat /etc/kernel/cmdline

# after editing that single line:
proxmox-boot-tool refresh

The first diagnosis is that the host is not busy, so it cannot be the host

This is the failure mode, and it costs an evening reliably.

The ticket says a guest is slow. The host CPU graph is flat at 25%. The storage is idle. Everyone concludes the host is fine and the application team is sent away to look at their code — where they find nothing, because there is nothing to find.

C-state exit latency is invisible to every graph in that paragraph. It does not consume CPU time, so utilisation does not move. It does not queue I/O, so the storage graphs stay clean. It adds tens to hundreds of microseconds to each wakeup, and a workload that wakes thousands of times per second turns that into whole milliseconds of added latency with no metric anywhere showing a resource under strain.

The tell is the shape of the problem rather than its size: latency is bad and throughput is fine, and the gap widens when the host is less busy rather than more, because an idle core goes deeper. If a slow guest speeds up when you put artificial load on the host, you have found it.

Lever 3 — transparent hugepages

A guest’s memory is backed by host pages, and every guest memory access walks two page tables — the guest’s own and the host’s EPT or NPT. Larger host pages mean shallower walks and fewer TLB entries burnt on the same amount of guest RAM, which is why THP is generally a win on a KVM host and why it is enabled by default.

The setting that goes wrong is not enabled. It is defrag.

Read-only / Safeboth THP settings, and whether they are being used— Read-only. AnonHugePages is how much memory is currently backed by transparent hugepages; on a busy KVM host this should be a large fraction of guest RAM. The bracketed value in each sysfs file is the active one.

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

grep -E 'AnonHugePages|HugePages_Total' /proc/meminfo

Read-only / SafeTHP on, and actually in use— Illustrative. The square brackets mark the active option. 'defer+madvise' is the setting that avoids synchronous compaction stalls while still letting the kernel build hugepages in the background.

# cat /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag

[always] madvise never
always defer [defer+madvise] madvise never

Illustrative output

defrag=always means a page fault that cannot immediately find a free hugepage will synchronously compact memory to build one. On a host whose memory is fragmented — which is any host that has been up for months running VMs of assorted sizes — that compaction can stall the faulting task for milliseconds. The symptom is periodic latency spikes with no corresponding CPU or I/O event, which is exactly the kind of thing that gets blamed on the storage.

defer+madvise is the safe production setting on a hypervisor: the kernel does not stall the fault, it wakes kswapd and khugepaged to do the work in the background, and applications that explicitly asked with madvise still get synchronous treatment.

Configuration changeset defrag to defer+madvise— Takes effect immediately for new allocations. Does not survive a reboot on its own — see the persistence section. Existing hugepage-backed memory is not disturbed.

echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag

cat /sys/kernel/mm/transparent_hugepage/defrag

Lever 4 — automatic NUMA balancing

On a multi-socket host, memory attached to the other socket costs more to reach — typically 1.5 to 2 times the local latency. Automatic NUMA balancing is the kernel periodically unmapping pages, catching the resulting faults, and migrating pages or tasks so they end up on the same node.

It is a good default for a host running ordinary VMs whose placement nobody has thought about. It is actively harmful in one specific case.

Read-only / Safetopology, current setting, and whether it is doing anything— Read-only. numastat per-node 'numa_miss' and 'numa_foreign' counters growing steadily means allocations are landing off-node. numa_balancing is 1 when enabled.

numactl --hardware

cat /proc/sys/kernel/numa_balancing

numastat
numastat -c qemu

The harmful case: a VM that has been given an explicit NUMA topology (qm set --numa 1) and pinned with --affinity has had its placement decided deliberately. Automatic balancing does not know that, and will keep sampling, faulting and migrating pages that were already in the right place. You pay the page-fault overhead for a migration that has nothing to gain.

Configuration changedisable automatic NUMA balancing— Only do this on a node whose latency-sensitive VMs have explicit numa and affinity set. On a node with unpinned guests this usually makes things worse, not better. Reversible by writing 1.

sysctl -w kernel.numa_balancing=0

sysctl kernel.numa_balancing

Lever 5 — swappiness

The Proxmox ZFS documentation recommends lowering vm.swappiness to 10 on servers. The default of 60 tells the kernel to be fairly willing to swap anonymous pages out in favour of keeping page cache. On a hypervisor the anonymous pages in question are guest RAM, and swapping guest RAM to disk produces the worst latency in the entire stack: the guest has no idea it happened and cannot compensate.

Configuration changelower swappiness— Takes effect immediately. Does not disable swap — it makes the kernel prefer reclaiming page cache over evicting guest memory. Keep swap configured; a host with no swap at all hits the OOM killer instead, which on a hypervisor means a guest is killed.

sysctl -w vm.swappiness=10

free -h
sysctl vm.swappiness

On a ZFS-root node there is an additional interaction worth knowing: the ARC is not page cache and is not reclaimed by the same path, so a node with an unbounded ARC and low swappiness can still end up with memory pressure that looks inexplicable. That is a storage-backend concern and the per-backend tuning lesson covers it.

Persisting all of it

Everything above is lost at the next reboot unless you write it down somewhere the boot process reads. Three mechanisms, three kinds of setting.

Configuration changesysctl settings — swappiness, NUMA balancing— Files in /etc/sysctl.d are applied at boot in lexical order. The --system flag re-applies everything now so you can confirm the file parses before you rely on it.

cat > /etc/sysctl.d/90-pve-host-tuning.conf <<'EOF'
vm.swappiness = 10
kernel.numa_balancing = 0
EOF

sysctl --system
sysctl vm.swappiness kernel.numa_balancing

Configuration changesysfs settings — THP defrag, governor— sysfs is not covered by sysctl.d. A small systemd unit ordered after multi-user.target is the durable way to write these, and unlike an rc.local script it is visible to systemctl status when it fails.

cat > /etc/systemd/system/pve-host-tuning.service <<'EOF'
[Unit]
Description=Proxmox host performance tuning
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag'
ExecStart=/usr/bin/cpupower frequency-set -g performance

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now pve-host-tuning.service
systemctl status pve-host-tuning.service --no-pager

Kernel command-line settings — C-state limits, IOMMU — go in /etc/default/grub plus update-grub on GRUB installs, or /etc/kernel/cmdline plus proxmox-boot-tool refresh on systemd-boot installs. proxmox-boot-tool status tells you which one this node is.

Verification that can fail

A tuning change you cannot prove did anything is a tuning change you should not have made. Each lever has a check that returns a value you can compare against the one you recorded first.

Read-only / Safethe after-state, in one block— Read-only. Run this before the change too, and diff the two. Every line here can disagree with what you intended, which is the point — 'the command exited 0' is not verification.

echo "== governor =="
grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | sort | uniq -c

echo "== achieved frequency =="
grep 'MHz' /proc/cpuinfo | sort -u -t: -k2 -n | tail -3

echo "== idle depth =="
cpupower idle-info | grep -E 'Available idle|Latency'

echo "== THP =="
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
grep AnonHugePages /proc/meminfo

echo "== numa =="
sysctl kernel.numa_balancing
numastat | head -6

echo "== pressure =="
cat /proc/pressure/cpu

The one that most often disagrees with expectation is the governor line. cpupower frequency-set applies to every online CPU, but a core that was offline at the time — or a node where the systemd unit ran before all cores were up — ends up with a mixed policy. sort | uniq -c across all CPUs catches that; reading cpu0 alone does not.

Under the hood

Why the frequency you asked for is not the frequency you get, and why that is not a bug.

The scaling_governor file is a request. On any recent server part the processor’s own power-management firmware makes the final decision, and it is subject to constraints the operating system cannot see or override: the package power limit (RAPL), the current thermal headroom, and — the one that surprises people most — how many cores are active and what instructions they are running.

This is why turbostat exists and why /proc/cpuinfo’s MHz field is worth reading even when you have already set the governor.

turbostat --quiet --show Core,CPU,Busy%,Bzy_MHz,PkgWatt,PkgTmp --interval 5

Bzy_MHz is the frequency achieved while the core was busy, which is the number the application experiences. Compare it against the hardware limits maximum from cpupower frequency-info. A large, persistent gap on a performance governor means the constraint is physical rather than policy, and no amount of sysfs writing will close it. The usual causes are an all-core turbo limit (correct and expected — all-core turbo is always below single-core turbo), a conservative power limit set in firmware, or AVX-512 frequency offsets on Intel parts, where running wide vector instructions drops the whole core’s clock by design.

That last one has a virtualisation twist. If one guest on a node runs an AVX-512-heavy workload, the frequency offset applies to the physical core — so a different guest sharing that core sees its clock drop for reasons entirely outside its own configuration. It is one of the harder noisy-neighbour cases to attribute, because the noisy guest may be using very little CPU time while doing it.

On the memory side, the reason THP matters more on a hypervisor than on a bare-metal server is the second level of translation. A guest virtual address becomes a guest physical address through the guest’s own page tables, and that becomes a host physical address through the EPT (Intel) or NPT (AMD). A full walk with 4 KiB pages at both levels is up to 24 memory references for one uncached translation. Backing guest memory with 2 MiB host pages removes an entire level from the host-side walk and cuts the TLB entries needed to map the same guest RAM by a factor of 512. That is why the win is measured in single-digit percentages on throughput and much more than that on memory-latency-bound guests, and why it is present even though nothing inside the guest knows about it.

What happens if you set every node in the cluster to the performance governor?

Three consequences, in the order you will meet them.

The power bill and the thermal envelope move immediately. On a two-socket node this is commonly 30 to 80 watts of extra continuous draw, and it does not drop when the node is idle — that is the whole point. In a rack provisioned close to its circuit limit, applying this to every node at once is a genuine capacity change and needs to be treated as one. Nodes also run hotter, fans run harder, and in a poorly-cooled room the result can be worse sustained performance than before, because thermal limits bite earlier.

Most guests will not notice. A steady-state workload already drives the demand-following governor to maximum frequency. The gain is concentrated in bursty, latency-sensitive guests, and if you cannot point to which ones those are, you have bought heat and nothing else.

Cluster consistency becomes the real argument for doing it anyway. This one cuts the other way. A cluster where node 1 is tuned and nodes 2 and 3 are not has guests that perform differently depending on where HA happened to place them — and a live migration can move a guest between those environments with no configuration change and no event in any log. That is a genuinely miserable class of ticket: the same VM, the same config, intermittently slower, correlating with nothing the operator can see.

So the reasonable answer is not “tune everything” or “tune nothing”. It is: decide per class of node, keep every node in a class identical, and make the class boundary something HA and the migration policy respect. If your latency-sensitive guests are pinned to a subset of nodes by an HA group, tune that subset and leave the rest. If they can land anywhere, you have to tune everywhere or accept the inconsistency — and tuning everywhere is the cheaper of those two.

Record the decision somewhere the next operator will find it. A node tuned for reasons nobody remembers gets reverted during the next troubleshooting session.

Common mistakes

Reading cpu0 and assuming the rest match. Governor and idle policy are per-CPU. Aggregate with sort | uniq -c across all of them.
Treating powersave under intel_pstate as the old cpufreq powersave. It is a demand-following policy, not a minimum-frequency pin, and it is a reasonable default.
Concluding the host is fine because utilisation is low. C-state exit latency and compaction stalls consume no CPU time and appear in no utilisation graph. Read PSI.
Leaving defrag=always on a long-uptime hypervisor. Synchronous compaction on a fragmented host produces millisecond latency spikes that get blamed on storage.
Disabling NUMA balancing on a node with unpinned guests. It helps there. Disable it only where explicit numa and affinity have already decided placement.
Confusing THP with qm set --hugepages. Different mechanisms, different costs; one is host-wide and opportunistic, the other is per-VM, pinned, and disables ballooning.
Tuning one node in a cluster. Live migration then makes performance depend on placement, with nothing in any log to explain it.
Not persisting sysfs settings. sysctl.d does not cover sysfs. The change survives until the next reboot and then quietly disappears.

Key takeaways

Utilisation is not saturation. A host at 30% CPU can still be the bottleneck; /proc/pressure/cpu is the metric that says so.
The governor request is not the achieved frequency. Bzy_MHz from turbostat is what the application experiences.
C-state exit latency is invisible to every utilisation graph and shows up as bad latency with fine throughput, worse when the host is less busy.
Cap C-states on the kernel command line to test the hypothesis; fix it in firmware once you have proved it.
THP is a win on KVM hosts because it shortens the two-level EPT/NPT walk. Set defrag=defer+madvise to get it without compaction stalls.
Disable automatic NUMA balancing only where explicit numa and affinity have already placed the guests.
Lower vm.swappiness to 10, but keep swap — a hypervisor with no swap reaches the OOM killer instead, and the OOM killer kills guests.
Persist sysctl in /etc/sysctl.d, sysfs in a systemd unit, and cmdline via the bootloader proxmox-boot-tool status names.
Tune per class of node and keep the class uniform, or live migration turns tuning into an unattributable performance mystery.

Knowledge check

Why this matters

Host tuning has a bad reputation, and it deserves part of it. Most of the “Proxmox performance tuning” advice in circulation is a list of sysctl values to paste in, with no measurement attached and no statement of what each one costs. Applied that way it is superstition, and it makes systems worse about as often as it makes them better.

What makes this lesson different is not the settings — they are the same four or five everyone names. It is that each one has a measurement that can contradict you, and a cost you can state. The governor has an achieved frequency you can compare against the hardware limit. C-states have an exit latency in microseconds you can compare against the application’s budget. THP has an AnonHugePages figure that says whether it is being used at all. NUMA balancing has counters that say whether allocations are landing off-node. If you cannot produce the number that motivated a change, you have not tuned anything, you have guessed.

The operational reason this matters more on a hypervisor than on a single-purpose server is attribution. On a dedicated database box, a latency problem has one plausible owner. On a node with thirty guests, a host-level cap manifests as thirty different application teams independently reporting that something is slow, none of whose evidence points at the host, because none of the host’s own graphs move. That is how a single misconfigured idle policy turns into six weeks of cross-team investigation.

The last thing worth carrying out of this lesson is the cluster consistency point. Tuning one node and not its neighbours makes performance a function of where HA happened to place a guest — and live migration can move it between the two environments silently, in the middle of the night, with no configuration change to point at. Uniform and untuned beats inconsistent and partly tuned, every time.