Containing a noisy neighbour · Proxmox VE

Somebody opens a ticket saying their VM is slow. Then a second person does. Then a third, from a different team, about an unrelated application. All three guests happen to live on the same node.

That shape — several unrelated guests degrading together on one host — is the signature of a noisy neighbour, and it is one of the few performance problems where the answer really is a configuration change on a single guest rather than a capacity purchase. Provided you can prove which guest, which is where most attempts at this go wrong.

Step 1: prove it is contention, not the node

Before hunting a guest, establish that guests are waiting for something. Utilisation will not tell you; pressure will.

Read-only / Safeis anything waiting, and for what?— Read-only. The three pressure files distinguish the resource under contention. 'some avg10' rising on cpu points at scheduling; on io it points at the storage path; on memory it points at reclaim.

cat /proc/pressure/cpu
cat /proc/pressure/io
cat /proc/pressure/memory

uptime
nproc

A load average well above nproc with CPU pressure climbing is CPU contention. I/O pressure climbing with CPU pressure flat is a storage queue. Memory pressure means reclaim, which is a different lesson.

This matters because the containment tool differs by resource, and applying a CPU limit to an I/O problem does nothing except make one team believe they were throttled for no reason.

Step 2: attribute it to a guest

Proxmox puts every guest in its own cgroup, which makes attribution mechanical rather than a guessing game.

Read-only / Safewhich guest is consuming the host?— Read-only. systemd-cgtop sorts live by cgroup, and PVE names each guest's scope qemu.slice/VMID.scope for VMs and lxc/VMID for containers, so the offender is identified by VMID directly.

systemd-cgtop --depth=3 -n 4

# the same question from the process side
pidstat -u -p ALL 2 3 | sort -k8 -rn | head -15

Read-only / SafeVMID 141 is the whole story— Illustrative. The qemu.slice scope names carry the VMID, so no correlation step is needed: 141 is using nearly seven cores while its neighbours use fractions of one.

# systemd-cgtop --depth=3 -n 1

Control Group                    Tasks   %CPU   Memory  Input/s Output/s
/                                  1204  812.4    198.2G        -        -
qemu.slice                          188  784.1    186.4G        -        -
qemu.slice/141.scope                 22  681.3     34.1G        -        -
qemu.slice/117.scope                 14   38.7     16.0G        -        -
qemu.slice/122.scope                 12   31.2     32.0G        -        -
lxc/205                               9    9.4      2.1G        -        -

Illustrative output

For I/O the equivalent attribution is per-cgroup byte counters.

Read-only / Safewhich guest is generating the I/O?— Read-only. io.stat under each guest's cgroup gives cumulative read and write bytes and IOPS per block device, keyed by major:minor. Sample it twice and subtract to get a rate.

# VMs
grep . /sys/fs/cgroup/qemu.slice/*.scope/io.stat 2>/dev/null

# containers
grep . /sys/fs/cgroup/lxc/*/io.stat 2>/dev/null

# and the pressure each guest is itself experiencing
grep . /sys/fs/cgroup/qemu.slice/*.scope/io.pressure 2>/dev/null

Step 3: choose the right lever

Proxmox exposes three distinct CPU controls and they are not interchangeable. The most common mistake in this whole subject is reaching for a hard cap when a weight was wanted.

Option	Type	Accepted	Default	Effect
`cpuunits`	Relative weight	`1 - 262144`, clamped to `1 - 10000`	`100` (cgroup v2)	Share of CPU when contended. No effect on an idle host.
`cpulimit`	Hard cap	`0 - 128`	`0` (none)	Total host CPU time in cores. `2.5` means 250%. Applies always.
`cores` / `sockets`	Topology	—	—	How many vCPUs exist at all. Changing it needs a guest restart.

The documentation describes cpuunits as “often called CPU shares or CPU weight… a relative weight”, and cpulimit as “a floating point value representing CPU time in percent, so 1.0 is equal to 100%, 2.5 to 250%”.

The cpuunits range has a wrinkle worth knowing. The qm man page documents 1 - 262144, but the API schema adds that the value “will be clamped to [1, 10000] in cgroup v2” — and PVE 8 and later run cgroup v2 exclusively. So a guest set to 50000 is silently running at 10000, and two guests set to 50000 and 200000 have the same weight as each other rather than a 4:1 ratio. Stay inside 1 - 10000 and the number you set is the number in effect.

The difference in practice:

cpuunits costs nothing when the host is not contended. A guest with cpuunits 50 on a quiet node runs exactly as fast as one with cpuunits 400. The weight only decides who yields when the host runs out of CPU. That makes it the safe, always-appropriate setting.
cpulimit applies even on an empty host. A guest capped at 2.0 cores can never exceed 200% of one core’s worth of host CPU time, at 3 a.m. on a node with 62 idle cores. That is sometimes what you want — chargeback, a test guest, a batch job you have promised will never interfere — and it is very often not.

Configuration changelower the weight of the noisy guest — the reversible first move— Applies live to a running VM; no restart, no interruption. Halving the weight relative to the default of 100 means this guest yields to its neighbours when the host is contended, and is unaffected when it is not.

VMID=141

qm set "$VMID" --cpuunits 50

qm config "$VMID" | grep -E 'cpuunits|cpulimit|cores'

Configuration changeor raise the weight of the victims instead— Applies live. Equivalent in effect and easier to justify politically, because no team is told their guest was throttled. Weight is relative, so raising some is the same as lowering others.

for VMID in 117 122; do
qm set "$VMID" --cpuunits 400
done

for VMID in 117 122; do
qm config "$VMID" | grep -E 'cpuunits'
done

Service impact possiblehard-cap a guest that must not exceed a share— Applies live, and applies unconditionally — including when the host is idle. Set this only when a guaranteed ceiling is the requirement. A cap below what the guest needs to complete its work will make it permanently slow rather than occasionally slow.

VMID=141

qm set "$VMID" --cpulimit 4

qm config "$VMID" | grep cpulimit

Containers use the same options through pct:

Configuration changethe container equivalents— Applies live to a running container. pct exposes the same cgroup controls with the same semantics as qm does for VMs.

CTID=205

pct set "$CTID" --cpuunits 50
pct set "$CTID" --cpulimit 2

pct config "$CTID" | grep -E 'cpuunits|cpulimit'

Step 4: I/O throttles

CPU weight does nothing for a guest saturating a storage backend. That needs a per-disk throttle, set on the disk parameter rather than on the guest as a whole.

Configuration changethrottle a single disk's bandwidth and IOPS— Applies live to a running VM. The mbps values are megabytes per second; the _rd and _wr variants set read and write independently, and the _max variants allow a burst above the sustained figure. Re-issue with the values removed to lift the throttle.

VMID=141

qm set "$VMID" --scsi0 \
local-zfs:vm-141-disk-0,iothread=1,mbps_rd=200,mbps_wr=100,iops_rd=5000,iops_wr=2000

qm config "$VMID" | grep scsi0

Step 5: verify from the cgroup, not the config file

qm config tells you what you asked for. The cgroup tells you what the kernel is enforcing. These disagree more often than you would like — after a failed live apply, after a migration, after a guest restart that picked up a pending change you had forgotten about.

Read-only / Safewhat is the kernel actually enforcing?— Read-only. cpu.weight is the translated cpuunits value; cpu.max is 'quota period' in microseconds, where 'max' means uncapped. io.max carries the throttle. These are the authoritative values.

VMID=141
CG="/sys/fs/cgroup/qemu.slice/$VMID.scope"

cat "$CG/cpu.weight"
cat "$CG/cpu.max"
cat "$CG/io.max" 2>/dev/null

# and the effect: how much throttled time has this guest accumulated?
grep -E 'nr_throttled|throttled_usec' "$CG/cpu.stat"

Read-only / Safethe cap is real, and the guest is hitting it— Illustrative. cpu.max of '400000 100000' is 4 cores' worth per 100 ms period. nr_throttled climbing means the guest is being stopped at the end of periods, which is the evidence that the cap is binding rather than merely set.

# cat /sys/fs/cgroup/qemu.slice/141.scope/cpu.max /sys/fs/cgroup/qemu.slice/141.scope/cpu.stat

400000 100000
usage_usec 8841200311
user_usec 7710455102
system_usec 1130745209
nr_periods 812204
nr_throttled 190338
throttled_usec 41882200914

Illustrative output

nr_throttled is the number that closes the loop. A cap that is set but never hit contributes nothing; a cap being hit constantly means the guest is permanently constrained, which is a different conversation from containing an occasional burst.

A hard cap turns a bursty guest into a permanently slow one, and it looks like a different fault

This is the failure this lesson exists to prevent.

A guest bursts to 12 cores for ninety seconds every hour while it runs a report. During those ninety seconds its neighbours suffer. An operator sets cpulimit 4 and the complaints stop, so the ticket is closed.

What actually happened: the report now takes four and a half hours instead of ninety seconds. The guest’s own team does not connect this to anything — nobody told them, and qm config is not something they can read — so they open a new ticket about their reporting job being slow. That ticket is investigated fresh, by someone else, who finds a guest at 100% CPU with no host contention anywhere, and concludes the application has a performance regression.

The evidence that identifies it takes one command: nr_throttled in the guest’s cpu.stat climbing steadily. A guest being throttled by its own cgroup looks, from inside the guest, exactly like a guest whose CPU is simply slow — the vCPU is stopped by the host and the guest kernel has no way to observe why. %steal inside the guest will show it, and %steal is routinely dismissed as “the host is busy” when in this case the host is idle.

Prefer cpuunits for containment. A weight has no effect on an idle host, so it cannot produce this failure at all: the guest gets its twelve cores at 3 a.m. and yields them at 10 a.m. when its neighbours need them. That is almost always the actual requirement, stated correctly.

Reserve cpulimit for cases where a guaranteed ceiling is genuinely the specification — chargeback, a contractual limit, a test guest that must not disturb production — and when you set one, tell the guest’s owner what it is.

When a limit is the wrong answer

Containment is a tourniquet. It is the right response to one guest behaving abnormally, and the wrong response to a node that is simply full.

The distinction is whether the sum of what the guests need fits.

Read-only / Safeis this a noisy neighbour, or an overloaded node?— Read-only. Sum the configured vCPUs across running guests and compare against physical cores. Substantial oversubscription plus sustained CPU pressure is a capacity problem that no per-guest limit will solve.

echo "physical cores: $(nproc)"

qm list | awk 'NR>1 && $3=="running" {print $1}' | while read -r VMID; do
qm config "$VMID" | awk -v id="$VMID" '/^cores:/ {c=$2} /^sockets:/ {s=$2} END {print id, (c?c:1)*(s?s:1)}'
done | awk '{t+=$2} END {print "configured vCPUs on running guests:", t}'

cat /proc/pressure/cpu

If the node is oversubscribed and every guest genuinely wants its allocation, throttling one guest just chooses a victim. The honest answers are to migrate guests off, add a node, or right-size the guests — and that is the xvii-performance-right-sizing conversation, informed by the N x u / (N - 1) arithmetic from Part II.

Set weights before you need them

The best time to configure cpuunits is when a guest is created, not during an incident. Three tiers is usually enough:

Tier	`cpuunits`	For
Latency-sensitive	400	Databases, brokers, anything user-facing
Standard	100	The default; most guests
Batch / best-effort	20	Reporting, builds, backups, test guests

With that in place, contention resolves itself in the direction you would have chosen anyway, at 3 a.m., without anyone being paged. A cluster where every guest is at the default weight has no policy at all — it has whichever guest asked first.

Under the hood

What cpuunits and cpulimit actually become, and why one can cause an outage while the other cannot.

Both land in the same cgroup v2 CPU controller, but on different files with fundamentally different semantics.

cpuunits becomes cpu.weight. The Linux CFS scheduler distributes CPU time among runnable cgroups in proportion to their weights. Weight is only consulted when there is a decision to make — that is, when more tasks are runnable than there are CPUs. On an uncontended host every runnable task is scheduled immediately and the weights are never consulted at all. This is why a weight change cannot slow a guest on an idle node: there is no mechanism by which it could.

The documented range is 1 - 262144, with a cgroup v2 default of 100. Note that PVE 8 and later run cgroup v2 exclusively; the old 1024 default belongs to cgroup v1 and you will still see it quoted in older documentation and forum posts.

cpulimit becomes cpu.max, expressed as quota period in microseconds. cpulimit 4 on the default 100 ms period gives 400000 100000: the cgroup may consume 400 ms of CPU time in each 100 ms of wall-clock time, which is four cores’ worth. When the quota is exhausted, every task in the cgroup is stopped until the period ends.

That last sentence is the whole risk, and its consequences are worth spelling out because they are not obvious from the description.

A VM’s vCPUs are threads in one QEMU process, in one cgroup. When the quota runs out, all of them stop simultaneously — including a vCPU that happens to be holding a spinlock inside the guest kernel. The other vCPUs, when they resume, spin waiting for a lock whose holder is descheduled and cannot run until the next period. The guest burns its fresh quota spinning, exhausts it again, and the effect compounds. This is the classic CFS bandwidth throttling pathology, and it is why a guest’s observed throughput under a quota can be far below the fraction the quota nominally allows. A guest capped at 50% of its cores does not generally run at 50% speed; it runs at rather less, and the gap widens with vCPU count.

I/O throttles are enforced somewhere else entirely. The mbps and iops parameters on a disk are implemented by QEMU’s own block-layer throttle, in userspace, on the I/O path for that specific drive — not by the cgroup io.max controller. Two practical consequences: they travel with the disk configuration across a live migration, and they apply to the virtual disk regardless of which host device backs it, so they survive a storage migration too. They also mean io.max in the guest’s cgroup will be empty even when a throttle is very much in force, which catches people out during verification. For VMs, qm config is the authority on I/O throttles; the cgroup is the authority on CPU.

What happens if the noisy guest is a backup job, or Ceph recovery, rather than a guest at all?

Then none of the tools in this lesson apply, and reaching for them wastes the first hour.

This case is common enough to be worth recognising by its shape: the host is contended, systemd-cgtop shows no guest consuming anything unusual, and the pressure is on I/O rather than CPU. The consumer is a host process, and there are three usual suspects.

A vzdump backup. Reading every block of every guest disk is exactly the workload that saturates a storage backend, and it runs at night when nobody is watching — until the window creeps into business hours or a retry starts one at 09:00. The containment is vzdump’s own bandwidth limit rather than a cgroup: a bwlimit in the backup job or in /etc/vzdump.conf, expressed in KiB/s. Check the running task first — pvenode task list will show a vzdump task in progress, which is usually faster than deducing it from I/O counters.

ZFS scrub or resilver. A scrub reads the entire pool. zpool status tells you immediately, including the estimated time remaining. A scrub can be paused with zpool scrub -p, which is the right move if it is genuinely hurting production, provided you let it resume later. A resilver should almost never be paused: it is the pool restoring redundancy after a disk failure, and the window where the pool is degraded is the window where a second failure loses data.

Ceph recovery or backfill. ceph -s shows it. This one is worth special mention because the advice most operators have absorbed for throttling it is out of date. On Squid and Tentacle the mClock scheduler is the default, and it overrides osd_max_backfills and osd_recovery_max_active and forces the osd_recovery_sleep family to zero — so setting them, which is what most guides still say, changes nothing at all. The current lever is the mClock profile: ceph config set osd osd_mclock_profile high_client_ops biases the scheduler toward client I/O for the duration. And as with a resilver, recovery is restoring redundancy — slowing it down lengthens the window in which another failure hurts, so it is a trade rather than a free win.

The general lesson: before containing a guest, confirm a guest is responsible. systemd-cgtop showing the host busy but no scope busy is the signal that the consumer is not a guest at all, and it takes four seconds to check.

Common mistakes

Reaching for cpulimit when cpuunits was wanted. A weight costs nothing on an idle host; a cap applies always and turns a bursty guest into a permanently slow one.
Trusting qm config as verification. The cgroup is what the kernel enforces. cpu.weight, cpu.max and nr_throttled are the truth.
Throttling the victim. The culprit shows high throughput and low pressure; the victims show low throughput and high pressure. Read io.stat and io.pressure together.
Setting an I/O ceiling at the guest’s normal requirement. A binding throttle produces guest task hangs and read-only remounts, not a gentle slowdown. Set it above the working figure.
Not telling the guest’s owner. A throttled guest looks, from inside, exactly like a slow one. The follow-up ticket gets investigated from scratch by someone who does not know.
Containing a guest on a node that is simply full. A limit then just chooses which team suffers. Compare configured vCPUs against physical cores before deciding.
Assuming the consumer is a guest. Backups, scrubs, resilvers and Ceph recovery are host-side and need entirely different levers.
Throttling Ceph recovery with osd_recovery_sleep on Squid or Tentacle. mClock is the default scheduler there and forces those settings to zero. Change the mClock profile instead.

Key takeaways

Establish contention with /proc/pressure/* before hunting a guest; utilisation cannot answer the question.
Attribute with systemd-cgtop — PVE names scopes by VMID, so the offender identifies itself.
cpuunits is a relative weight, default 100, and has no effect on an uncontended host. It is the safe first move. Keep values inside 1 - 10000; cgroup v2 clamps anything higher.
cpulimit is a hard cap in cores, range 0 - 128, default 0, and applies unconditionally. Use it only when a guaranteed ceiling is the actual requirement.
I/O throttles go on the disk parameter and are enforced by QEMU, not by the cgroup — so io.max stays empty even when they are active.
Verify from cpu.weight, cpu.max and nr_throttled, not from qm config.
CFS quota stops every vCPU at once, so a throttled multi-vCPU guest loses more performance than the quota fraction suggests.
Assign weights at guest creation in tiers. A cluster where everything is at the default has no contention policy.
A limit on a full node just picks a victim. Check configured vCPUs against physical cores first.

Knowledge check

Why this matters

Noisy-neighbour containment is one of the few performance interventions where the tool is genuinely simple and the judgement is genuinely hard. qm set --cpuunits 50 is not a difficult command. Knowing that it is the right command, on the right guest, for a problem that is actually contention rather than capacity, is where the work is.

The pattern worth internalising is that a limit is a statement about priority, not about performance. cpuunits does not make anything faster or slower in absolute terms — it decides who wins when the host cannot satisfy everyone. That framing makes the design question obvious: which of your guests should lose, and by how much? A cluster where every guest carries the default weight has never answered that question, so the answer is decided at 3 a.m. by whichever workload happened to ask first. Three tiers assigned at creation time is not sophisticated, and it is enormously better than nothing.

The second thing to carry away is a general one about hidden state. A cpulimit applied during an incident is invisible to the team that owns the guest, invisible in every dashboard they can see, and indistinguishable from slow hardware from inside the guest. It will outlive the incident, the ticket, and quite possibly the operator who set it. Any control with that combination of properties — durable, invisible to the affected party, and mimicking a different fault — needs a record attached to it: a note in the VM description, a ticket reference, a review date. This is the same discipline as documenting a apt-mark hold or a pinned kernel, and it fails in the same way when skipped.

Finally: containment buys time, and time is what you need to do the real work. It is a legitimate move at 3 a.m. It stops being legitimate around the third week, when the throttle has quietly become the architecture and nobody has reopened the question of whether the node is simply full.