Diagnosing "the VM is slow" · Proxmox VE

Why this matters in production

“VM is slow” is the most common production ticket. Operators must triage in minutes, not hours.

The diagnostic flow

flowchart LR
  A[VM is slow] --> B[Where is the latency?]
  B --> C[Guest CPU/memory]
  B --> D[Storage I/O]
  B --> E[Network]
  B --> F[Application]
  B --> G[Database]
  B --> H[Other VMs noisy neighbour]

Step 1: Measure inside the guest

top && free -h && iostat -xz 1 3

High %steal → host CPU oversubscribed.
High memory pressure → memory overcommit.
High disk await → storage bottleneck.

Step 2: Check the host

uptime && top -bn1 | head -20

If all CPUs are saturated, the host is the bottleneck. Identify the noisy neighbour with pidstat or top.

Step 3: Check storage

Read-only / Safe— Check storage layer.

iostat -xz 1 3 && zpool iostat 1 2 2>/dev/null || ceph osd perf 2>/dev/null

Look for:

High await.
OSD nearfull.
ZFS fragmentation.
Ceph recovery in progress.

Step 4: Check network

Read-only / Safe— Check network errors and drops.

for nic in $(ls /sys/class/net | grep -v lo); do echo "=== $nic ==="; ethtool -S $nic 2>/dev/null | grep -E 'errors|dropped' | head -3; done

Step 5: Check the application

If the infrastructure looks fine, the problem is the application. Common causes:

Database query gone wrong (missing index, full table scan).
Bug introduced by recent deployment.
External dependency slow (API upstream).

Check application logs.

Utilisation is not saturation

The five steps above all read utilisation, and utilisation cannot answer the question they are being asked. A CPU at 30 % is not a CPU with 70 % of its performance in reserve — it is a CPU that spent 30 % of wall-clock time executing something, at whatever speed it happened to be running, with no statement about how long anything waited to get there.

Pressure Stall Information is the metric that says whether anything is waiting, and PVE 9 exposes it per guest as well as per host.

Read-only / Safeis anything actually stalled, and which guest?— Read-only. 'some avg10' is the share of the last ten seconds in which at least one task was stalled on that resource. Non-zero CPU pressure on a host reporting low utilisation is the signature that utilisation is lying to you.

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

# per guest, from the cluster API
pvesh get /cluster/resources --type vm --output-format json | jq -r '
.[] | select(.status=="running")
| [.vmid, .name, .pressurecpusome, .pressureiosome] | @tsv'

Read it alongside the guest-side numbers rather than instead of them. High %steal inside the guest plus high host CPU pressure is genuine contention. High %steal with the host idle and no pressure is a cpulimit throttling the guest — a different fault with an identical symptom, covered in xvii-performance-noisy-neighbours.

The host is not busy, so it cannot be the host

This is the conclusion that closes a slow-VM ticket wrongly, and it is reached by following exactly the steps above.

The guest is slow. Host CPU is flat at 25 %, storage is idle, network is clean. Every measurement says the infrastructure is fine, so the ticket goes back to the application team, who find nothing — because there is nothing to find.

Three host-level faults produce precisely this evidence, and none of them consume a resource that any utilisation graph reports:

Deep C-states. Waking a core from C6 costs on the order of a hundred microseconds. A guest that wakes thousands of times a second turns that into whole milliseconds of added latency, and the effect is worse when the host is less busy, because an idle core goes deeper.
Synchronous THP compaction. With defrag=always on a fragmented long-uptime host, a page fault can stall for milliseconds building a hugepage. It looks like a storage spike and is not.
A frequency governor that never ramps. The achieved clock, not the governor name, is what the application experiences.

The shape to recognise is bad latency with fine throughput, and no resource showing strain. xvii-performance-host-tuning covers the measurement for each. Before returning a ticket to an application team, it is worth ten seconds to check /proc/pressure/cpu and the achieved CPU frequency — if pressure is non-zero on a quiet host, the host is implicated whatever its utilisation says.

The diagnostic tree

flowchart TB
  A[VM is slow] --> B[VM CPU high?]
  B -->|yes| C[App or guest bottleneck]
  B -->|no| D[Storage latency high?]
  D -->|yes| E[Storage bottleneck]
  D -->|no| F[Network errors?]
  F -->|yes| G[Network issue]
  F -->|no| H[Application/database issue]

A break/fix exercise

Break/Fixadvanced30 minperformance

Database VM slow after Ceph cluster expansion

Symptoms

Database VM 100 reports slow queries (5-10s instead of 100ms)
Other VMs on the same cluster are unaffected
Ceph health is HEALTH_OK; no OSD issues

Available evidence

ceph osd df tree shows utilisation across all OSDs increased after expansion
Database VM uses replication=3; Ceph pool nearfull at 75%
ceph -s shows active backfill; recovery is consuming bandwidth and IOPS
ceph config get osd osd_op_queue returns mclock_scheduler, and osd_mclock_profile returns balanced

Show diagnosis & remediation

Root cause

Recent cluster expansion triggered a large backfill operation. Backfill is consuming significant IOPS and network bandwidth, slowing the database VM. The other VMs (web servers) have caching that masks the IOPS degradation.

Safe remediation

Bias the scheduler toward client I/O: ceph config set osd osd_mclock_profile high_client_ops. Do NOT set osd_recovery_sleep or osd_recovery_max_active — mClock has been the default scheduler since Reef and overrides osd_max_backfills and the osd_recovery_max_active family while forcing the osd_recovery_sleep options to zero, so those settings are stored and reported back but have no effect. Once backfill completes (verify via ceph -s), return the profile to balanced.

Verification

ceph config get osd osd_mclock_profile returns high_client_ops. Database VM query latency returns to normal. ceph -s shows recovery still progressing at a lower rate. After backfill completes, the profile is set back to balanced and confirmed.

Prevention

Schedule cluster expansions during maintenance windows. Note that slowing recovery is a durability trade rather than a free win: until backfill completes some placement groups are below the pool's replica count, so a second failure in that window is more expensive. Decide deliberately which side of that trade the situation is on.

Production considerations

What happens if several unrelated guests on one node degrade at the same time?

That coincidence is itself the diagnosis, and it changes the order of the five steps entirely.

A single slow guest is most often that guest’s own problem: its application, its configuration, its disk. Several unrelated guests, on one node, degrading together, is not a coincidence — it is a shared resource, and continuing to investigate any one of them individually is the mistake.

Confirm the boundary first. Do the affected guests share a node, a storage backend, or a network path? pvesh get /cluster/resources --type vm gives node placement in one line. If the set of affected guests maps exactly onto one node, the node is implicated. If it maps onto a storage name across several nodes, the backend is.

Then find the consumer, which may not be a guest at all. systemd-cgtop --depth=3 sorts live by cgroup, and PVE names each guest’s scope by VMID. One scope dominating is a noisy neighbour. No scope dominating while the host is loaded means the consumer is a host process — a vzdump backup that ran long, a ZFS scrub or resilver, Ceph recovery — and none of the per-guest levers apply. pvenode task list answers the backup question faster than any I/O counter.

Distinguish culprit from victim. Both look slow. The culprit shows high throughput with low pressure, because it is getting what it asks for; the victims show low throughput with high pressure, because they are queued behind it. Reading io.stat and io.pressure together across the node’s cgroups settles it in one pass, and it is worth doing before throttling anything, because throttling the victim is a real and embarrassing outcome.

And check whether the node is simply full. Sum the configured vCPUs of the running guests against nproc. If every guest genuinely wants its allocation and the node is oversubscribed, there is no noisy neighbour — there is a capacity problem, and a limit applied to one guest only chooses which team suffers.

The general point: the first question in a slow-VM ticket is not “what is wrong with this guest” but “how many guests are affected, and what do they share”. Answering it takes under a minute and it redirects the whole investigation.

Common mistakes

Optimising storage when the issue is the application. Measure each layer before changing any of it.
Restarting things instead of measuring. A restart destroys the evidence and frequently fixes the symptom for long enough that the cause is never found.
Blaming the network without checking. ethtool -S for errors and drops takes seconds.
Concluding the host is fine because utilisation is low. C-state exit latency and THP compaction stalls consume no measurable resource. Read /proc/pressure/*.
Investigating one guest when several are affected. The overlap is the diagnosis.
Throttling the victim rather than the culprit. High throughput with low pressure identifies the culprit.
Throttling Ceph recovery with osd_recovery_sleep. mClock is the default scheduler on Squid and Tentacle and forces it to zero. Set osd_mclock_profile instead.
Assuming %steal means the host is busy. It also appears when a cpulimit throttles the guest on an idle host.

Key takeaways

Measure from the guest outward, but establish first how many guests are affected and what they share.
Utilisation is not saturation. /proc/pressure/cpu, /proc/pressure/io and the per-guest pressurecpusome and pressureiosome fields say whether anything is actually waiting.
Bad latency with fine throughput, on a host showing no strain, points at host-level causes: C-states, THP compaction, frequency ramp.
%steal on an idle host means a cpulimit, not contention.
The culprit has high throughput and low pressure; the victims have the reverse.
A host that is busy with no busy cgroup means the consumer is a host process — backup, scrub, resilver or Ceph recovery.
On Squid and Tentacle, bias recovery with osd_mclock_profile; the legacy sleep and max-active settings are overridden and do nothing.
Slowing recovery lengthens the degraded window, so it is a durability trade rather than a free win.
Build a knowledge base of past slow-VM tickets; any ticket taking more than thirty minutes should produce a runbook entry.