Container resource limits under cgroup v2 · Proxmox VE

A VM with 4 GiB of RAM has 4 GiB of RAM. Its kernel manages that memory, runs out of it in the ordinary way, and its own OOM killer resolves the situation inside the guest where the guest’s logs record it.

A container with memory: 4096 has a cgroup limit of 4 GiB against the host’s memory. Everything about how it behaves at the boundary is different, and so is where you look afterwards.

PVE 9 runs pure cgroup v2 - the unified hierarchy, no v1 fallback - so this lesson describes one mechanism rather than two.

CPU: three knobs that are not the same knob

Option	Range	What it does	Enforced how
`cores`	1 - N	How many CPUs the container sees	cpuset - which host CPUs its tasks may run on
`cpulimit`	0 - 8192, default 0	Ceiling on total CPU time consumed	cgroup CPU bandwidth quota
`cpuunits`	0 - 500000, default 100 under cgroup v2	Relative share when CPUs are contended	cgroup CPU weight

The distinction that matters in practice:

cpulimit is a hard ceiling and applies always. cpulimit: 2 means the container gets at most two CPUs’ worth of time per second even if the host is completely idle.
cpuunits is a weight and applies only under contention. A container with cpuunits: 50 against one with cpuunits: 200 gets a quarter of the contested time - but if nothing else wants CPU, it can use the whole machine up to its cpulimit.
cores changes what the workload believes. Applications size their thread pools, worker counts and GC threads from the CPU count they observe. A container with cores: 2 and cpulimit: 8 will still start two workers.

Configuration changeset CPU limits on a running container— Changes take effect immediately for cpulimit and cpuunits; a change to cores takes effect on restart for applications that read the count at startup.

set -euo pipefail
CTID=200

pct set "$CTID" --cores 4 --cpulimit 2 --cpuunits 100

pct config "$CTID" | grep -E '^(cores|cpulimit|cpuunits):'

Memory: the limit is a wall, not a warning

memory sets the cgroup memory maximum in MiB - documented range 16 to N, default 512. swap sets an additional swap allowance, default 512 MiB.

Configuration changeset memory and swap— Lowering a memory limit below current usage on a running container can trigger reclaim or an immediate OOM kill inside it. Raise freely; lower during a maintenance window.

set -euo pipefail
CTID=200

pct set "$CTID" --memory 4096 --swap 512

# Find the container's cgroup on the host. The parent directory name has
# differed between LXC versions, so discover it rather than hard-coding it.
CG=$(find /sys/fs/cgroup -maxdepth 2 -type d -name "$CTID" \
     -not -path '*monitor*' | head -1)
printf 'cgroup: %s\n' "$CG"

# Read back what the cgroup actually enforces, in bytes.
cat "$CG/memory.max" "$CG/memory.swap.max" "$CG/memory.current"

The container that keeps running and loses a process

The symptom. A container is up. systemctl inside it reports every unit active except one, which is in failed state with Killed. Or worse: the application is a multi-process server and one worker vanished, so the service is up and 1-in-8 requests time out. Container-level monitoring - is it pingable, is the port open - stays green throughout.

What happened. The container reached memory.max. The kernel’s cgroup OOM killer selected the largest process in that cgroup and killed it. The container’s other processes were untouched, so nothing restarted, nothing crashed, and no shutdown occurred.

Why the first diagnosis is wrong. Everyone looks in the container’s logs first, and the container’s logs contain nothing useful. There is no kernel inside the container to log the kill, and journalctl inside it sees only that a unit died. free inside the container may even show memory available, for reasons covered below. So the investigation goes down the application path - a bug, a bad request, a dependency - when the evidence is one level up.

Where the evidence actually is:

CTID=200
CG=$(find /sys/fs/cgroup -maxdepth 2 -type d -name "$CTID" -not -path '*monitor*' | head -1)
dmesg -T | grep -iE 'oom|killed process' | tail -20
journalctl -k --since '2 hours ago' | grep -i oom
cat "$CG/memory.events"

memory.events is the decisive one. Its oom_kill counter is a running total for that container’s cgroup, and a non-zero value is proof, with no log retention to worry about:

Read-only / Safethe counter that settles the argument

# cat /sys/fs/cgroup/lxc/200/memory.events   # path discovered as above

low 0
high 0
max 4127
oom 3
oom_kill 3

Illustrative output

Read that as: the container hit its hard limit 4127 times, went into OOM three times, and three processes were killed. A max count in the thousands with oom_kill at zero is also worth knowing about - it means the container is constantly reclaiming at its ceiling, which is a performance problem disguised as a working system.

lxcfs, and why free inside a container tells the truth

/proc/meminfo is a kernel interface describing the machine. A container has no machine of its own, so by default a process inside one reading /proc/meminfo gets the host’s numbers - a container limited to 4 GiB on a 512 GiB host would see 512 GiB, and free, top, the JVM’s default heap sizing and every “use 80% of RAM” heuristic would be wrong by two orders of magnitude.

PVE ships lxcfs, a small FUSE filesystem that synthesises a per-container view of /proc/meminfo, /proc/cpuinfo, /proc/stat, /proc/uptime, /proc/diskstats, /proc/swaps and /sys/devices/system/cpu/online from the container’s cgroup values, and bind-mounts those over the real ones inside the container.

That is why free -m inside a PVE container reports the container’s limit rather than the host’s, and it is a genuinely useful piece of engineering that most operators never notice because it just works.

Two things follow that do matter:

It is not universal. lxcfs virtualises a specific list of files. Anything reading cgroup values directly, or reading a file not on that list, sees host reality. A monitoring agent inside the container may report host CPU count while nproc reports the container’s.
If lxcfs is not running, the masks are gone. The container does not fail; it silently starts believing it has the whole host. An application that sizes a cache or a thread pool from that belief will then be killed by the cgroup limit it never knew about. systemctl status lxcfs on the host is worth including in node health checks precisely because its failure is invisible from inside.

PID limits, and the fork bomb question

A container that can create unlimited processes can exhaust the host’s global PID space, which affects every other container and the host itself. The cgroup v2 pids controller bounds this.

PVE does not expose a pids.max option on pct set, so it is set through the raw LXC configuration:

Configuration changecap the number of processes a container may create— Appends a raw LXC setting to the container config. Requires a restart. Set it well above the container's steady-state process count.

set -euo pipefail
CTID=200

CG=$(find /sys/fs/cgroup -maxdepth 2 -type d -name "$CTID" \
     -not -path '*monitor*' | head -1)

# Look at the steady state first - the limit must clear it comfortably.
cat "$CG/pids.current"

printf 'lxc.cgroup2.pids.max: 2048\n' >> "/etc/pve/lxc/$CTID.conf"
pct reboot "$CTID"

cat "$CG/pids.max"

What a container limit cannot do

Being explicit about the gaps is more useful than listing the knobs.

Resource	Bounded per container?	Notes
CPU time	Yes	`cpulimit` (quota), `cpuunits` (weight)
Memory	Yes	`memory`, `swap` - hard limits, OOM at the boundary
Processes	Yes	via raw `lxc.cgroup2.pids.max`
Block I/O	Partially	cgroup v2 `io` controller; PVE has no `pct` option for it
Network bandwidth	Yes, per interface	`rate=` on the `net[n]` option
Kernel memory structures	No	dentries, inodes, socket buffers are largely shared
Filesystem cache	No	Page cache is global; a container doing heavy I/O evicts others’ cache
Kernel time	No	A container issuing pathological syscalls costs host CPU in ways `cpulimit` accounts for poorly

That last block is the honest limit of the model. cgroups bound what a container consumes; they do not bound what a container costs. A container within every one of its limits can still degrade its neighbours through the page cache and the shared kernel, and no amount of limit-setting changes that. When a workload’s noisy-neighbour behaviour has to be bounded rather than merely discouraged, that is an argument for a VM.

What happens if you lower a running container's memory limit below its current usage?

The write to memory.max succeeds - the kernel accepts the new limit - and then it must reconcile reality with it.

First the kernel tries reclaim: dropping page cache, writing back dirty pages, swapping if memory.swap.max allows. If usage can be brought under the new limit that way, the container survives with a cold cache and a period of poor performance while it re-warms. This is the common case for a modest reduction and it looks, from outside, like the container got slow for ten minutes.

If reclaim cannot get there - because the memory is anonymous and in use - the cgroup OOM killer runs immediately, against the largest process. On a database container that is the database.

So the operational rule is asymmetric, and worth stating as a rule:

Raising a memory limit is safe and takes effect at once. Do it whenever.
Lowering one is a service-affecting operation. Check memory.current first, lower in steps, and do it in a window.

CTID=200
CG=$(find /sys/fs/cgroup -maxdepth 2 -type d -name "$CTID" -not -path '*monitor*' | head -1)
CUR=$(cat "$CG/memory.current")
printf 'current usage: %s MiB\n' "$((CUR / 1024 / 1024))"

If that number is above the limit you were about to set, you are not resizing a container, you are choosing which process dies.

Verification

Read-only / Safewhat the container is actually limited to, from both sides— Read-only. Comparing the host's cgroup view against the container's own view is how you catch an lxcfs problem or a limit that did not apply.

set -euo pipefail
CTID=200
CG=$(find /sys/fs/cgroup -maxdepth 2 -type d -name "$CTID" \
     -not -path '*monitor*' | head -1)

echo '--- enforced by the host ---'
printf 'memory.max   %s\n' "$(cat "$CG/memory.max")"
printf 'memory.current %s\n' "$(cat "$CG/memory.current")"
printf 'cpu.max      %s\n' "$(cat "$CG/cpu.max")"
printf 'cpu.weight   %s\n' "$(cat "$CG/cpu.weight")"
printf 'pids.max     %s\n' "$(cat "$CG/pids.max")"
grep oom_kill "$CG/memory.events"

echo '--- believed by the container ---'
pct exec "$CTID" -- sh -c 'nproc; free -m | head -2'

The two halves should agree. nproc reporting the host’s CPU count while cpu.max shows a quota is the signature of an lxcfs problem, and it means every application inside that container has sized itself for a machine it does not have.

Knowledge check

Why this matters

The reason container limits deserve their own lesson, rather than a table in the configuration lesson, is that they fail differently from VM limits and almost everyone’s instincts are calibrated on VMs.

A VM that runs out of memory has an event: the guest kernel logs it, the guest OOM killer runs, the guest’s own monitoring sees it. A container that runs out of memory has a counter incrementing on the host, a process missing, and a service that reports itself healthy. The evidence lives one level above where the symptom appears, in files most operators have never opened.

That asymmetry is the thing to internalise. When something inside a container behaves inexplicably - a worker that vanished, a fork that failed, an application that sized itself for a machine ten times larger than it has - the first move is to step out of the container and read the cgroup. The container’s own view is a synthesis, and when the synthesis and the enforcement disagree, the enforcement wins.