KSM, memory overcommit and the ballooning interaction

Memory is the resource that decides how many guests fit on a node, and it is the one you cannot bluff. A node short of CPU runs slowly. A node short of RAM invokes the OOM killer, and on a hypervisor the OOM killer kills a guest.

Proxmox gives you three mechanisms for fitting more guest memory onto a host than the host physically has, and they are frequently confused with one another:

This lesson is mostly about the second, because it is the one whose economics are least visible, and because its savings are the ones most often mistaken for capacity.

What KSM does

ksmd walks the anonymous memory of processes that have marked regions mergeable — on a Proxmox host, that is QEMU’s guest RAM — looking for pages whose contents are byte-for-byte identical. When it finds two, it maps both to a single physical page, marks it read-only, and frees the other. If either guest later writes to that page, a copy-on-write fault splits them apart again.

Twenty guests running the same Debian 13 image with the same packages share an enormous amount: kernel text, libc, the page cache of files they all read. Twenty guests running unrelated workloads with different distributions share almost nothing.

KSM’s savings are therefore a property of your workload mix, not of your configuration. Which is why the only useful thing to say about it is: measure it on your own node.

Measure it

# is it even running?
cat /sys/kernel/mm/ksm/run

# the raw counters
grep . /sys/kernel/mm/ksm/pages_shared      /sys/kernel/mm/ksm/pages_sharing      /sys/kernel/mm/ksm/pages_unshared      /sys/kernel/mm/ksm/pages_volatile      /sys/kernel/mm/ksm/full_scans

# the saving, in a unit a capacity plan can use
SHARING=$(cat /sys/kernel/mm/ksm/pages_sharing)
echo "saved: $(( SHARING * 4096 / 1024 / 1024 )) MiB"

# grep . /sys/kernel/mm/ksm/pages_*

/sys/kernel/mm/ksm/pages_shared:412883
/sys/kernel/mm/ksm/pages_sharing:4193021
/sys/kernel/mm/ksm/pages_unshared:1882401
/sys/kernel/mm/ksm/pages_volatile:318772

The two counters people mix up:

• pages_shared — how many distinct physical pages are being shared. This is memory KSM is still using.
• pages_sharing — how many additional references point at those pages. This is memory KSM has freed.

The saving is pages_sharing × 4096. In the capture above that is about 16 GiB — a real number on a 256 GiB node, and a number you can put in a sentence to a finance team.

Two more worth watching:

• pages_volatile counts pages that change too fast to be worth placing in KSM’s trees. Every one of them was scanned, hashed and discarded. A high pages_volatile with a low pages_sharing is precisely the “all cost, no benefit” case.
• full_scans increments each time ksmd has walked everything. If it is climbing fast, ksmd is working hard.

ps -o pid,comm,etime,time -C ksmd

# how aggressively is it currently scanning?
cat /sys/kernel/mm/ksm/pages_to_scan
cat /sys/kernel/mm/ksm/sleep_millisecs

When does it run? ksmtuned decides

Proxmox ships the ksm-control-daemon package, whose ksmtuned service turns KSM on and off based on memory pressure. It does not run KSM continuously, and understanding its rule is what makes the behaviour predictable.

From ksmtuned(8), the daemon “periodically estimates memory pressure from running qemu processes and adjusts Linux KSM aggressiveness by writing to sysfs controls”. Every KSM_MONITOR_INTERVAL it sums the RSS of running QEMU processes, compares available memory against a threshold, and either starts KSM with a boosted scan rate or stops it.

With the defaults, a 256 GiB node starts merging when available memory drops below roughly 51 GiB, and ramps pages_to_scan up by 300 each minute while the pressure persists, to a ceiling of 1250.

systemctl status ksmtuned --no-pager
cat /etc/ksmtuned.conf 2>/dev/null || echo "no config file: using defaults"

Ballooning is a different thing, and they compose

The most common confusion in this area is treating ballooning and KSM as alternatives. They address different memory.

Ballooning works from inside the guest. The virtio_balloon driver allocates pages within the guest, which makes the guest’s own kernel reclaim page cache and free memory, and then hands those pages back to the host. It reclaims memory the guest was not using. Proxmox enables the ballooning device by default because, in the documentation’s words, “it delivers useful information such as how much memory the guest really uses” — the device is worth having for the telemetry alone even when you never inflate it.

When a VM has a minimum memory lower than its maximum, the documentation describes the behaviour: Proxmox “will make sure that the minimum amount you specified is always available to the VM, and if RAM usage on the host is below a certain target percentage, will dynamically add memory to the guest up to the maximum memory specified”.

KSM works from outside the guest, on memory the guest is using and believes it has exclusively.

So they stack: ballooning shrinks the guest’s footprint to what it actually needs, and KSM then deduplicates what is left. On a fleet of similar guests both are worth having.

Excluding one guest from KSM

You do not have to choose between “KSM everywhere” and “KSM nowhere”. PVE exposes a per-VM boolean, allow-ksm, documented as “allow memory pages of this guest to be merged via KSM (Kernel Samepage Merging)”, defaulting to enabled.

VMID=100

qm set "$VMID" --allow-ksm 0

qm config "$VMID" | grep -E 'allow-ksm|hugepages|shares'

This is the right tool for a guest that has a reason not to be deduplicated — a database whose latency budget cannot absorb copy-on-write faults, a guest belonging to a different tenant, or one whose memory is known to be unique so scanning it is pure cost — on a node where KSM is otherwise earning its keep.

There is a related per-VM memory control that is easy to confuse with cpuunits: shares, documented as “amount of memory shares for auto-ballooning… the larger the number is, the more memory this VM gets. Number is relative to weights of all other running VMs”, defaulting to 1000, with zero disabling auto-ballooning for that guest. Auto-ballooning is performed by pvestatd, so this is a memory weight and has nothing to do with CPU weight despite the similar shape.

KSM and hugepages work against each other

This is the interaction that surprises people, and it is a direct consequence of how KSM finds duplicates.

KSM compares pages. Its unit is the base page — 4 KiB on x86-64. A 2 MiB transparent hugepage is not a candidate for merging as a unit, and for KSM to merge anything inside it, that hugepage has to be split back into 512 base pages first.

So on a host where both are active, khugepaged is assembling base pages into hugepages while ksmd is splitting hugepages apart to merge their contents. The two are undoing each other’s work, and both are spending CPU to do it.

Explicit hugepages — qm set VMID --hugepages 2 or 1024 — are worse still from KSM’s point of view: that memory is pinned, not mergeable, and KSM cannot touch it at all. It is also not balloonable.

Overcommit ratios that survive a failure

Here is the part that turns a tuning topic into a capacity topic.

Suppose a three-node cluster, 256 GiB per node. Guests on each node are configured for 340 GiB of maximum memory. Steady state is comfortable: ballooning has them down to about 210 GiB actually resident, KSM is saving another 16 GiB, and each node sits at roughly 76% memory utilisation. Everything is green.

Now node 3 fails and HA restarts its guests on nodes 1 and 2.

The arithmetic is the one the Linux course’s N+1 lesson gives:

per-node load after one failure = N x u / (N - 1)

With N = 3 and u = 76%, the survivors land at 3 x 76 / 2 = 114%. The 66.7% ceiling for a three-node cluster was breached long before the failure; nobody noticed because nothing was wrong until something was.

And memory has a property CPU does not: the overcommit mechanisms do not scale with the emergency. Three things go wrong at once.

• Ballooning cannot reclaim what is in use. The guests were already ballooned down to their working set. There is no slack left to take.
• KSM’s savings do not double. The restarted guests are new QEMU processes with fresh memory that has not been scanned. ksmd needs full scans — minutes to tens of minutes on a large node — before sharing rebuilds. During the failover, KSM is contributing close to nothing.
• Swap is the only remaining valve, and swapping guest RAM makes the node so slow that HA fencing timers start to look plausible.

TOTAL_KIB=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
AVAIL_KIB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
SAVED_KIB=$(( $(cat /sys/kernel/mm/ksm/pages_sharing) * 4 ))

echo "physical:        $(( TOTAL_KIB / 1024 )) MiB"
echo "available now:   $(( AVAIL_KIB / 1024 )) MiB"
echo "KSM is saving:   $(( SAVED_KIB / 1024 )) MiB"
echo "available if KSM stopped: $(( (AVAIL_KIB - SAVED_KIB) / 1024 )) MiB"

If that last line is negative, the node cannot survive losing KSM, and losing KSM is a thing that happens.

Turning it off, if you decide to

There are two ways, and choosing the wrong one on a tight node is an outage.

systemctl stop ksmtuned
systemctl disable ksmtuned

echo 0 > /sys/kernel/mm/ksm/run

cat /sys/kernel/mm/ksm/run
grep . /sys/kernel/mm/ksm/pages_sharing

SAVED_MIB=$(( $(cat /sys/kernel/mm/ksm/pages_sharing) * 4096 / 1024 / 1024 ))
AVAIL_MIB=$(( $(awk '/MemAvailable/ {print $2}' /proc/meminfo) / 1024 ))
echo "this will consume ${SAVED_MIB} MiB; ${AVAIL_MIB} MiB is available"

# only if that arithmetic leaves a comfortable margin:
echo 2 > /sys/kernel/mm/ksm/run

The kernel documentation is explicit about the difference: run is “set to 0 to stop ksmd from running but keep merged pages”, versus “set to 2 to stop ksmd and unmerge all pages currently merged”.

Almost every operational reason to stop KSM — it is using too much CPU, you are moving the node to a hugepages policy, you are investigating whether it is implicated in a fault — is served by 0. Use 2 only when you specifically need the memory un-deduplicated, and only after checking that the node can absorb it.

Common mistakes

• Reading pages_shared as the saving. It is the memory KSM is still using. pages_sharing is the memory it freed.
• Counting KSM savings as capacity. They evaporate during patch windows and failovers — exactly when you need them.
• Running KSM on heterogeneous guests. Full scan cost, near-zero sharing. Check pages_sharing against total guest RAM; under ~5% it is not paying for itself.
• Writing 2 to run on a tight node. That unmerges everything immediately and can OOM the host. Write 0 unless you specifically need the memory split apart.
• Combining KSM with explicit hugepages on the same node. Hugepage-backed guest memory is pinned and unmergeable; THP-backed memory has to be split before KSM can merge it. Pick one policy.
• Assuming ballooning works because it is configured. It needs virtio_balloon inside the guest. A guest without the driver never returns anything.
• Enabling KSM across tenant boundaries. The copy-on-write timing difference is an information leak between guests.

Key takeaways

• KSM merges identical anonymous pages and splits them again on write. Its benefit is a property of your workload mix, not of configuration.
• pages_sharing × 4096 is the memory saved. pages_shared is not. pages_volatile high with pages_sharing low is all cost, no benefit.
• ksmtuned starts KSM when available memory falls below KSM_THRES_COEF percent of total — 20% by default — and ramps pages_to_scan by 300 per minute to a ceiling of 1250.
• Ballooning reclaims memory the guest is not using, from inside; KSM deduplicates memory the guest is using, from outside. They compose.
• KSM and hugepages fight: ksmd splits what khugepaged assembles, and explicit hugepages are pinned and unmergeable. Choose per node.
• qm set VMID --allow-ksm 0 excludes one guest from merging without disabling KSM for the node. shares is the separate per-VM weight for auto-ballooning, not for CPU.
• Plan capacity against the un-deduplicated figure. N x u / (N - 1) still governs the failover, and KSM contributes nothing during one because restarted guests arrive unscanned.
• echo 0 > run stops merging safely; echo 2 > run unmerges and can OOM an overcommitted host.
• KSM is for guests in one trust domain. The copy-on-write timing difference leaks information across guests.

Knowledge check