VM right-sizing: matching resources to workloads

Most VMs in production are over-sized. A 16 GB VM that uses 3 GB is wasting 13 GB. With 100 such VMs, you’re burning 1.3 TB of RAM that could host 30+ additional workloads.

This lesson shows how to measure actual workload needs and right-size VMs without sacrificing safety margin.

The default sizing trap

PVE’s default VM creation provides generous CPU and memory — usually 1 core and 1 GB minimum. For a real workload, neither is enough. So admins bump up to “production-sized”: 4 cores, 8 GB, 100 GB disk.

The result: most workloads run with low utilisation. A web server peaks at 1.5 cores and 4 GB of RAM but is allocated 4 cores and 8 GB. The unused 2.5 cores and 4 GB of RAM could host more VMs on the same hardware.

The cost of over-provisioning is invisible but real:

Less density per host (fewer VMs per host)
More hosts needed (more licensing, power, cooling)
Slower live migration (more memory to transfer)
Larger backup windows (more data per VM)

The sizing workflow

A right-sizing pass for one VM:

Measure current utilisation — run for at least 7 days at peak workload
Profile memory usage patterns — distinguish RSS (actual use) from cache and buffer
Add safety margin — 50% for RAM, 25% for CPU
Apply and monitor — verify no degradation
Re-tune after 30 days

Measuring CPU utilisation

Use the qemu-guest-agent from inside the VM, or node_exporter from the host:

# Host-side (via PVE API)
pvesh get /nodes/pve-01/qemu/100/status/current
# Look at "cpu" (current usage 0.0–1.0) and "cpus" (allocated count)

# Guest-side
top
# Look at %CPU column under load
mpstat 1 5
# 1-second samples, 5 iterations

A VM’s CPU utilisation is reported as a fraction of one core. A 4-core VM at 100% utilisation shows as “cpu”: 4.0 in the PVE API.

For a web server under typical load:

Idle: 0.05–0.10 (5–10% of one core)
Average: 0.20–0.40
Peak (cache miss, request burst): 0.80–2.0

Right-sizing: aim for peak ≤ 75% of allocated cores. So if peak is 2.0, allocate 3 cores.

Measuring memory utilisation

# Inside the VM
free -h
vmstat 1 5
# Look at:
#   - "si/so" (swap in/out) — should be 0 or near-0
#   - "bi/bo" (block I/O) — high values mean memory pressure
#   - "cache" — OS uses free RAM for cache; that's normal

# More detailed
ps aux --sort -rss | head
# Top processes by RSS

# Tracking over time (collect into a file)
for i in $(seq 1 60); do
  free -m | awk '/^Mem:/ &#123;print $3&#125;' >> /tmp/mem-used.log
  sleep 60
done

The numbers to focus on:

RSS (resident set size) — actual physical memory used by processes. This is what you care about.
Cache — OS uses free RAM as page cache. Don’t count this as “used” — it’s released automatically when apps need it.
Swap in/out — non-zero swap means the VM is under memory pressure and is hurting itself.

A web server under typical load:

RSS: 1.5–2.5 GB
Cache: variable, often equal to or larger than RSS
Swap in/out: 0

Right-sizing RAM: aim for peak RSS × 1.5. So if peak RSS is 2 GB, allocate 3 GB.

Disk sizing

Disk sizing is straightforward — measure current usage, add growth margin, project over time.

# Inside the VM
df -h
du -sh /var /home /opt /tmp 2>/dev/null

# Host-side (for thin-provisioned disks)
pvesh get /nodes/pve-01/qemu/100/config \
  --output-format json | jq '.rootfs'

# Or via the GUI: VM → Hardware → Hard Disk → "Disk size" vs "Disk usage"

For thin-provisioned disks, allocate based on growth forecast, not current usage. A VM using 20 GB today will likely need 100 GB in two years. Allocate 200 GB thin-provisioned; the storage backend only allocates what’s actually used.

For thick-provisioned disks (rare), allocate current usage plus forecast.

Ballooning for dynamic sizing

The qemu guest agent’s balloon driver (virtio-balloon) allows the host to reclaim memory from a VM that’s not using it, and give it back when needed.

# Enable ballooning on a VM
qm set 100 --balloon 0   # 0 = ballooning disabled
qm set 100 --balloon 50  # 50% minimum memory (1 GB on a 2 GB VM)

The VM’s qemu-guest-agent reports its actual memory usage to the host. If the VM is using 800 MB of a 2 GB allocation, the host can inflate the balloon to reclaim up to 1.2 GB.

Ballooning is great for:

Mixed workloads where some VMs are idle at any given moment
Cloud-like elasticity on a fixed pool
Cost-sensitive deployments

Ballooning is bad for:

Performance-sensitive workloads (balloon pressure adds latency)
VMs without qemu-guest-agent installed

Set balloon to 0 (disabled) for databases and other latency-sensitive workloads.

CPU hot-plug

For VMs that need CPU dynamically:

# Allocate a CPU pool
qm set 100 --cores 4 --sockets 1

# Hot-plug to 6 cores (if the guest supports it)
qm set 100 --cores 6
# The guest kernel sees the new cores immediately

# Hot-unplug back to 4
qm set 100 --cores 4

Hot-plug works for most modern Linux kernels (CPU hot-plug is well supported) but not for Windows without integration services.

For RAM hot-plug:

qm set 100 --memory 4096
# Most Linux guests auto-online the new memory; some need:
# echo 1 > /sys/devices/system/memory/memoryXXX/online

For production VMs that don’t need to dynamically scale, use fixed sizes. Hot-plug adds complexity.

Sizing templates by workload type

Common sizing patterns:

Workload	CPU	RAM	Disk	Notes
Reverse proxy / load balancer	1 core	1 GB	10 GB	Network I/O bound, scale horizontally
Web server (static)	2 cores	2 GB	20 GB	Cache-friendly, may need more disk for content
Web server (dynamic)	2 cores	4 GB	50 GB	DB calls, sessions
App server (Java)	4 cores	8 GB	30 GB	JVM heap typically 4 GB
App server (Go / Rust)	2 cores	1 GB	20 GB	Lean runtime
Database (Postgres)	4 cores	16 GB	100 GB	shared_buffers = 25% RAM
Database (Redis)	4 cores	8 GB	50 GB	All-in-memory, plan for dataset size
PBS server	4 cores	8 GB	2 TB	Chunk storage, garbage collection matters
CI runner	4 cores	8 GB	50 GB	Bursty load

These are starting points. Always measure in production and adjust.

Right-sizing at scale

For a fleet of 100+ VMs, manual sizing doesn’t scale. Use:

Capacity planning tools (VMPatrol, vCenter) — though PVE doesn’t have an equivalent built in
Grafana dashboards showing per-VM peak utilisation
Ansible / config management to apply sizing rules
Quarterly reviews of the top 10 over-provisioned VMs

The metric to track: average utilisation / average allocation. Aim for 60–70% — under 50% means over-provisioning; over 80% means under-provisioning.

Common mistakes

Sizing for the busiest day of the year. Most workloads have peaks. A web server that hits 3 cores on Black Friday doesn’t need 3 cores year-round. Use auto-scaling (more VMs) or scheduled right-sizing.
Allocating RAM as “I might need it later”. Memory that’s allocated to a VM is unavailable to others. Allocate what the workload needs today, with margin for predictable growth.
Ignoring NUMA. On multi-socket hosts, a VM that crosses NUMA boundaries has memory access latency penalties. Pin the VM to a single NUMA node (qm set 100 --numa 1 with appropriate topology).
Forgetting balloon and hot-plug costs. Both add a small amount of CPU overhead. For performance-critical VMs, disable them.

Production considerations

Over-commit carefully. PVE supports CPU over-commit (host has 16 cores, VMs allocated 32 cores total) but not RAM over-commit by default. Track per-host RAM committed vs available.
Sizing is iterative. Right-size, monitor for 30 days, adjust. Don’t expect to get it right on the first pass.
Document your sizing decisions. A spreadsheet or wiki with “VM X is sized 2 cores / 4 GB because workload Y peaks at Z” is invaluable when you’re asked to justify costs or plan capacity.

Key takeaways

Measure first, then size. Aim for peak utilisation ≤ 75%.
Use balloon for elastic workloads, disable for databases.
Right-sizing is iterative — measure, apply, monitor, adjust.
Track utilisation, not allocation, to detect over-provisioning.