Disk I/O paths: aio, cache, iothread and how they interact · Proxmox VE

Proxmox disk tuning advice circulates as isolated recommendations: use cache=writeback, enable iothread, set aio=native. Each of those is correct in some situation, none is correct universally, and two of them constrain each other in a way that is not signposted anywhere in the interface.

The previous lesson covered the controller choice, cache modes and IO threads. This one adds the parameter that is usually left at its default — aio — and then treats the four as one system, because that is what they are.

The four parameters

scsi0: <storage>:vm-118-disk-0,aio=<native|threads|io_uring>,
       cache=<directsync|none|unsafe|writeback|writethrough>,
       iothread=<1|0>,discard=<on|ignore>

Parameter	Decides
`aio`	How the host submits I/O to the kernel
`cache`	Whether the host page cache is involved, and when the guest is told a write completed
`iothread`	Which thread does the work
`discard`	Whether guest TRIM reaches the storage

aio: three ways to submit I/O

Value	Mechanism	Requires `O_DIRECT`	Notes
`threads`	A pool of worker threads issues ordinary blocking system calls	No	Works with every cache mode. Highest per-request overhead
`native`	Linux kernel AIO (`libaio`)	Yes	Low overhead, but only truly asynchronous on `O_DIRECT` file descriptors
`io_uring`	The modern `io_uring` submission and completion ring	No	Asynchronous with or without `O_DIRECT`, fewer system calls per request

native and page-cached modes are a bad combination, and nothing warns you

This is the constraint that makes aio worth a lesson.

Linux kernel AIO is only genuinely asynchronous on file descriptors opened with O_DIRECT. On a page-cached descriptor, a submission that has to touch the page cache blocks the submitting thread — silently, without an error, and with no indication in any Proxmox interface.

Which produces the compatibility rule:

`aio`	`cache=none`	`cache=directsync`	`writeback` / `writethrough` / `unsafe`
`native`	Good	Good	Avoid — submission can block
`threads`	Fine	Fine	Fine
`io_uring`	Good	Good	Good

cache=none and cache=directsync are the modes that use O_DIRECT. Everything else routes through the host page cache. So aio=native belongs with the first two and nowhere else — and the combination that appears most often in copied-and-pasted advice, aio=native with cache=writeback, is exactly the one to avoid.

cache: what the guest is told, and when

The documentation frames this as a signalling question rather than a performance one:

Setting the Cache mode of the hard drive will impact how the host system will notify the guest systems of block write completions. The No cache default means that the guest system will be notified that a write is complete when each block reaches the physical storage write queue.

Mode	Host page cache	Write acknowledged when	Survives a host crash
`none` (default)	Bypassed	The block reaches the physical storage write queue	Yes, if the storage honours flushes
`directsync`	Bypassed	The block is on stable storage	Yes
`writethrough`	Used for reads	The block is on stable storage	Yes
`writeback`	Used	The block is in the host page cache	Only data the guest flushed
`unsafe`	Used, and flush requests are ignored	Immediately	No

Double caching is the reason writeback is not free on ZFS or Ceph

writeback puts the host page cache in front of the storage. On a backend that already caches — ZFS with its ARC, Ceph with its own caching layers, a SAN with a battery-backed controller cache — the same data is now held in two places, and the host page cache is competing for the RAM the ARC wanted.

On ZFS in particular this shows up as ARC pressure that degrades performance for every guest on the pool, in exchange for improving one. cache=none is the default for good reasons on those backends, and moving away from it should be a measured decision rather than a general improvement.

Where writeback genuinely earns its place is on a plain directory or LVM storage with no caching layer of its own, and on a host with power protection you actually trust.

iothread: which thread does the work

The documented behaviour and its constraint:

The option IO Thread can only be used when using a disk with the VirtIO controller, or with the SCSI controller, when the emulated controller type is VirtIO SCSI single. With IO Thread enabled, QEMU creates one I/O thread per storage controller rather than handling all I/O in the main event loop or vCPU threads.

Two things follow, and the second is the one people miss.

The controller requirement is absolute. iothread=1 on a disk attached to a plain VirtIO SCSI controller does nothing useful. The controller type must be VirtIO SCSI single, which is the type that gives each disk its own controller — and that is what makes “one I/O thread per controller” mean “one per disk”.

It is a concurrency change, not a speed change. Without it, disk I/O completion competes with the guest’s vCPU threads and QEMU’s main event loop. That contention is what an iothread removes. So the benefit appears on guests with several busy disks and several vCPUs under simultaneous load, and is close to zero on a single-disk guest doing sequential work.

Configuration changethe arrangement that actually enables iothreads— Requires a cold start. Changing the controller type on an existing guest can change device enumeration inside it - test on one guest before doing it in bulk.

set -euo pipefail
VMID=118

# VirtIO SCSI single is the prerequisite, not an optimisation on top.
qm set "$VMID" --scsihw virtio-scsi-single

qm set "$VMID" --scsi0 ceph-vm:vm-118-disk-0,iothread=1,cache=none,aio=io_uring,discard=on

qm shutdown "$VMID" --timeout 300
qm start "$VMID"

Read-only / Safewhat the running machine actually uses— Resolved values, not configured ones. Read this before and after any tuning change.

# qm showcmd 118 --pretty | grep -E 'aio=|cache=|iothread|drive'

-drive 'file=rbd:ceph-vm/vm-118-disk-0:conf=/etc/pve/ceph.conf,if=none,id=drive-scsi0,cache=none,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap'
-device 'virtio-scsi-single,id=scsihw0,bus=pci.0,addr=0x5,iothread=iothread-virtioscsi0'

Illustrative output

Measuring, rather than adopting

A tuning change with no measurement is a rumour with a change ticket attached. The measurement does not have to be elaborate, but it does have to run inside the guest, with a working set larger than the host page cache, and it has to be repeated on the unchanged configuration.

Service impact possiblea baseline that is not lying to you— Run INSIDE the guest, on a test guest, on a filesystem you can afford to fill. It writes several GiB and will saturate the storage while it runs.

set -euo pipefail

# 4 KiB random writes, queue depth 32, direct I/O to bypass the guest cache.
# The size must exceed the host page cache or you are measuring RAM.
fio --name=randwrite \
  --filename=/var/tmp/fio-test \
  --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 \
  --size=8G --runtime=120 --time_based \
  --direct=1 --ioengine=libaio --group_reporting

# The number that matters for most workloads is not IOPS - it is the
# 99th percentile completion latency, which fio reports as clat percentiles.

rm -f /var/tmp/fio-test

Sensible defaults, and when to leave them

Backend	`cache`	`aio`	`iothread`	`discard`
Ceph/RBD	`none`	`io_uring`	`1` with `virtio-scsi-single`	`on`
ZFS (local)	`none`	`io_uring`	`1` with `virtio-scsi-single`	`on`
LVM-thin	`none`	`io_uring`	`1` with `virtio-scsi-single`	`on`
Directory on local SSD	`none`, or `writeback` if power-protected	`io_uring`	`1` with `virtio-scsi-single`	`on`
NFS	`none` or `writeback`	`io_uring` or `threads`	`1` with `virtio-scsi-single`	`on` for qcow2
Disposable build guest	`unsafe`	any	any	`on`

The honest summary of that table is that cache=none plus io_uring plus iothread on virtio-scsi-single is right almost everywhere, and the interesting decisions are all about when to deviate. If a tuning discussion starts from anywhere else, ask what measurement produced it.

What happens if you set aio=native with cache=writeback because a benchmark said it was fastest?

The benchmark was probably not wrong, and it was probably measuring the host page cache.

Under a light, cache-friendly workload the configuration performs well, because most of the I/O never reaches storage. As the working set grows past what the page cache can hold, submissions increasingly have to touch storage — and on a page-cached descriptor, kernel AIO submission blocks the submitting thread while it does.

The symptom is a guest that is fast when idle and develops latency spikes under load, correlated with nothing visible on the host: CPU is not saturated, the storage is not saturated, and the guest is waiting. It looks like a storage problem and is a submission-path problem.

It is also the hardest class of tuning bug to find, because the configuration was chosen deliberately, tested, and found to be good. The evidence that identifies it is a latency histogram from inside the guest under sustained load, compared against the same guest with cache=none — and the comparison has to be like for like, because the two configurations have completely different cache behaviour.

Prevention is cheaper: keep native with O_DIRECT modes only, and prefer io_uring, which is asynchronous either way and removes the constraint from the decision.