Storage I/O performance · Proxmox VE

Why this matters in production

Storage is the most common performance bottleneck. Diagnosing it correctly is the difference between a quick fix and a long outage.

It is also the layer most often blamed wrongly, because every slow thing feels like a slow disk from inside a guest. The discipline in this lesson is to measure at each layer and let the numbers say which one owns the problem.

The four numbers

(Recap from Part V.) For every storage workload, know:

IOPS (operations per second)
Latency (time per operation)
Throughput (bytes per second)
Queue depth (concurrent in-flight I/Os)

These are not four independent measurements. Three of them are related, and the relationship is the single most useful thing to know when reading a benchmark:

IOPS = queue depth / latency

A device at 1 ms latency serving one request at a time does 1,000 IOPS. The same device at 1 ms latency with 32 requests in flight does 32,000. So a benchmark reporting a big IOPS number tells you nothing until you know the queue depth it used — and a vendor figure quoted at queue depth 256 describes a workload almost no guest generates.

The corollary matters more. If your workload is single-threaded and synchronous — a database committing a transaction, a filesystem writing its journal — its queue depth is 1, and only latency determines its speed. Adding faster devices with more parallelism does not help it at all. This is why “we moved to all-NVMe and the database is the same speed” is a recognisable outcome.

Where the bottleneck lives

flowchart TB
  A[Application write] --> B[Guest kernel + virtio-scsi]
  B --> C[QEMU]
  C --> D[Host block layer]
  D --> E[Storage layer]
  E --> F[Physical device]

Each step can be the bottleneck. Measure at each step.

Guest-side measurement

Read-only / Safewhat does the guest think its disk is doing?— Read-only, run inside the guest. The -x flag gives the extended columns that actually answer the question; without it you get throughput only.

iostat -xz 1 5

Read-only / Safeiostat -xz, and the three columns that matter— Illustrative. r_await and w_await are the average latency in milliseconds as the guest experiences it; aqu-sz is the average queue depth; %util is the share of time at least one request was outstanding.

# iostat -xz 1

Device  r/s     w/s    rMB/s  wMB/s  r_await  w_await  aqu-sz  %util
sda     412.0  1880.0   6.44   29.38     0.71    18.42   35.12   99.60

Illustrative output

Reading that capture:

w_await of 18.4 ms against r_await of 0.71 ms. Writes are 26 times slower than reads. That asymmetry points at the write path specifically — replication, a full cache, or a device whose write latency collapses under sustained load.
aqu-sz of 35 means 35 requests are outstanding on average. The guest is asking for far more parallelism than it is getting back.
%util at 99.6% is the column most often misread. On a device that can service many requests concurrently — every SSD, every NVMe, every SAN LUN — %util says only that something was in flight, not that the device is saturated. An NVMe drive at 100% %util may be at 5% of its capability. Use latency and queue depth to judge saturation; %util is only meaningful for a single-queue spinning disk.

Host-side measurement

Read-only / Safethe same question, one layer down— Read-only. Run on the host and compare against the guest capture. Latency present at the host is a host or storage problem; latency present only in the guest is above the host block layer — QEMU, the guest kernel, or a throttle.

iostat -xz 1 5

cat /proc/pressure/io

The comparison is the point. Three cases, three different investigations:

Guest latency	Host latency	Owns the problem
High	High	The storage backend or the device
High	Low	QEMU, an I/O throttle, or the guest’s own configuration
Low	High	Another guest, or a host process — backup, scrub, recovery

ZFS

Read-only / Safeper-vdev latency and throughput— Read-only. The -v flag breaks the pool down by vdev and device, which is how you find the one disk dragging a mirror. -w gives latency histograms instead of averages, which is where tail latency hides.

zpool iostat -v 1 3

zpool iostat -w 1 1

arcstat 1 3

An average latency figure hides the tail, and the tail is what a user experiences. -w prints a histogram: a pool whose average is 2 ms but which has a bucket at 500 ms is a pool with a problem, and its average will never say so.

Ceph

Read-only / Safeper-OSD latency, and whether recovery is running— Read-only. osd perf reports commit and apply latency per OSD. One OSD far above its peers is usually a failing device rather than a cluster-wide problem, and it drags every operation that touches it.

ceph osd perf

ceph -s
ceph osd pool stats

Benchmarking with fio

fio is the standard Linux storage benchmark. Run it on the host to measure storage-level performance, and inside the guest to measure what the guest actually gets.

Service impact possibleallocate a scratch volume and benchmark it— Creates a temporary volume on the target storage, benchmarks it, and removes it. This consumes real IOPS on shared storage, so it will affect running guests — run it in a window, not during an incident.

STORAGE=local-zfs
SCRATCH=vm-9999-disk-0

pvesm alloc "$STORAGE" 9999 "$SCRATCH" 32G
TARGET=$(pvesm path "$STORAGE:$SCRATCH")
echo "benchmark target: $TARGET"

fio --name=randread4k --filename="$TARGET" \
  --ioengine=libaio --direct=1 --bs=4k --rw=randread \
  --iodepth=32 --numjobs=4 --runtime=30 --time_based \
  --group_reporting

Destructiveclean up the scratch volume— Removes the volume and everything on it. Confirm the volume name is the scratch one and not a guest disk before running this — pvesm free does not prompt.

STORAGE=local-zfs
SCRATCH=vm-9999-disk-0

pvesm list "$STORAGE" | grep 9999
pvesm free "$STORAGE:$SCRATCH"

Benchmark with the workload pattern

A benchmark with the wrong block size, queue depth, or sync semantics produces meaningless numbers. The four parameters that decide whether a result means anything:

--bs — 4k for database random I/O, 64k–1M for sequential and backup traffic. A 4k result does not predict a 1M workload or the reverse.
--iodepth — 1 for a synchronous single-threaded workload, 32 or more for something parallel. This is the parameter that makes vendor numbers look impressive.
--rw — and the read/write ratio. randrw with --rwmixread=70 is closer to most real workloads than either pure pattern.
--fsync or --sync — a database commits with a flush on every transaction. A benchmark that does not is measuring a different thing entirely, often by an order of magnitude.

Always benchmark with the actual workload pattern, and record the full command line alongside the result. A number without its command line is not reproducible and is therefore not a baseline.

VM disk tuning

Common optimisations:

Setting	Effect
`iothread=1`	Gives the disk its own I/O thread instead of sharing the main loop
`discard=on`	Lets the guest return freed blocks to thin storage
`ssd=1`	Advertises the disk as non-rotational to the guest
`virtio-scsi-single`	Modern paravirtualised controller, one controller per disk
`aio=io_uring`	Default on PVE 9; lower syscall overhead than the older engines

virtio-scsi-single and iothread=1 belong together: the -single variant gives each disk its own controller, which is what makes a per-disk I/O thread possible. Setting iothread=1 on a shared virtio-scsi controller does considerably less.

Cache modes, by what they do to flushes

The cache mode table is usually presented as a performance ranking. It is better understood as two independent questions: does the host page cache get used, and what does the guest disk cache advertise.

Mode	Host page cache	Guest disk cache	Notes
`none`	Disabled	writeback	PVE default. Balances performance and safety, better writes
`writethrough`	Read cache only	writethrough	Better reads; an fsync for each write
`writeback`	Read and write	writeback	Faster; relies on the guest sending flushes
`directsync`	Disabled	writethrough	Safest and slowest; an fsync for each write
`unsafe`	Read and write	writeback	Ignores flush commands from the guest

`unsafe` is the only one that discards your data integrity

Four of these five modes honour the guest’s flush commands. A guest that calls fsync before acknowledging a transaction gets a real durability guarantee under none, writethrough, writeback and directsync — they differ in how much they cache and how much they cost, not in whether they lie.

unsafe is different in kind. The Proxmox documentation states it plainly: it “ignores flush commands from the guest”, giving “no data integrity even if the guest is sending flush commands”, and is “not recommended for production use”.

Its legitimate use is an installation or template build you are prepared to redo — a case where the entire disk is disposable and the speed is worth it. Its illegitimate use is setting it on a database guest during a performance investigation and forgetting, which produces a guest that appears to have durable commits and does not, until a host crash proves otherwise.

If you set unsafe for a build, put the removal in the same change ticket.

Production considerations

Under the hood

Why the same physical device gives a guest a fraction of what the host measures.

A guest write does not go to a disk. It traverses a stack, and each layer adds latency and can add serialisation.

The guest application writes to the guest page cache. The guest kernel’s block layer queues it and hands it to the virtio-scsi driver, which places a descriptor in a shared ring buffer and signals the host. QEMU picks it up — either on its main event loop, or, with iothread=1, on a thread dedicated to that disk — and issues the corresponding host I/O through the configured aio engine. The host block layer queues that against the backing device, which may be a zvol, an RBD image, an LVM volume or a file, each of which has its own translation to perform before anything reaches hardware.

Three consequences that explain most of the surprises.

Without iothread, disks share a thread. QEMU’s main loop handles device emulation for the whole VM. A guest doing heavy I/O on one disk can delay I/O on another, and can delay unrelated device emulation, on the same VM. This is why iothread=1 with virtio-scsi-single is the standard recommendation and why its benefit appears mainly on guests with several busy disks or many concurrent requests.

A flush is a barrier, and barriers do not parallelise. When the guest issues a cache flush, everything queued must complete before the flush is acknowledged. A workload that flushes frequently — a database, a journalled filesystem with barrier=1 — converts a deep, parallel queue into a series of round trips. Its effective queue depth is 1 no matter what the device supports, which is why such workloads are latency-bound and why the --fsync parameter changes an fio result by an order of magnitude.

Replication multiplies the write, and the acknowledgement waits for all of it. On a size=3 Ceph pool the primary OSD must receive acknowledgements from both replicas before the client’s write completes. The client’s latency is therefore the slowest of the three, plus network round trips. This is the mechanism behind the “one slow OSD” effect, and it is also why Ceph write latency has a floor set by network round-trip time that no faster drive can lower.

The practical synthesis: when guest latency exceeds host latency, the gap is in the layers between them, and the usual suspects are a missing iothread, a flush-heavy workload meeting a deep-queue-optimised backend, or an I/O throttle on the disk. When guest and host latency match, the problem is at or below the host block layer and the guest configuration is not where to look.

What happens if you move a database guest from local NVMe to Ceph and it gets slower?

This is common, it is not a misconfiguration, and understanding why prevents a long and fruitless tuning exercise.

The measurements will look contradictory. The Ceph cluster benchmarks higher than the local NVMe on aggregate IOPS and throughput. ceph osd perf shows healthy latencies. ceph -s is HEALTH_OK. And the database’s transaction rate has halved.

The mechanism is latency, not capacity. A local NVMe write completes in tens of microseconds. A Ceph write goes over the network to the primary OSD, out to two replicas, back, and only then completes — commonly a millisecond or more even on a well-built cluster. That is an order of magnitude, and for a workload whose queue depth is 1 because it flushes on every commit, throughput is exactly 1 / latency. Ten times the latency is a tenth of the transaction rate, and no amount of aggregate cluster capacity changes it.

Why the aggregate benchmark misled you. fio at --iodepth=32 measures how much the cluster can do in parallel, which Ceph is genuinely good at. The database is not asking for parallelism; it is asking for one fast round trip, repeatedly. The benchmark answered a question nobody asked.

What actually helps, in order of effect. Reduce the round trips: a lower-latency cluster network — 25 GbE rather than 10, and RDMA where available — moves the floor. Reduce the number of commits: application-level batching, group commit, or a longer commit_delay in PostgreSQL converts many small synchronous writes into fewer larger ones, which is by far the largest lever and belongs to the application team. And reconsider the placement: a database with a hard latency requirement may simply belong on local NVMe with replication handled at the database layer rather than the storage layer, which is a legitimate architecture and not a defeat.

What does not help. More OSDs, faster drives, a bigger cluster, or tuning the guest’s disk parameters. All of them add parallelism to a workload that cannot use it.

The general lesson. Before migrating a workload between storage backends, characterise it by queue depth and flush frequency rather than by IOPS. A high-queue-depth workload moves to distributed storage happily. A synchronous, low-queue-depth one is buying network round trips, and the decision should be made knowingly.

Common mistakes

Reading %util as saturation. On any device that services requests concurrently it means only that something was in flight. Use latency and queue depth.
Quoting an IOPS figure without its queue depth and block size. The number is meaningless without them.
Benchmarking with a write pattern against a live guest disk. It overwrites the guest’s filesystem with no prompt.
Comparing a vendor benchmark at queue depth 256 with a database at queue depth 1. They describe different machines.
Setting iothread=1 without virtio-scsi-single. The per-disk controller is what makes the per-disk thread useful.
Treating cache=unsafe as a performance option. It discards flush commands and is the only mode that removes the guest’s durability guarantee.
Optimising the wrong layer. Compare guest latency against host latency first; the answer tells you where to work.
Enabling discard=on and assuming TRIM works. The guest must run fstrim, and every layer beneath must pass the discard through.
Averaging away the tail. zpool iostat -w shows the histogram; the average never reveals a 500 ms bucket.

Key takeaways

IOPS = queue depth / latency. A synchronous, flush-heavy workload has queue depth 1 and is bound purely by latency; parallelism cannot help it.
Measure at each layer and compare. Guest latency high with host latency low means the problem is between them, not in the storage.
%util is not saturation on a concurrent device. Latency and queue depth are.
Use zpool iostat -w and ceph osd perf to find the tail and the outlier; one slow OSD slows every write to every PG it belongs to.
Benchmark against an allocated scratch volume, never a live guest disk, and record the full fio command line with the result.
Four of the five cache modes honour guest flushes; unsafe does not, and it is the only one that removes a durability guarantee.
cache=none is the PVE default and is the right starting point.
virtio-scsi-single plus iothread=1 belong together.
TRIM needs the disk option, the guest timer, and a storage layer that passes discards through.