IOPS, latency, queue depth, sync vs async

Why this matters in production

“The storage is slow” tickets are nearly always diagnosable with four numbers: IOPS, latency, throughput, and queue depth. Knowing what each measures — and what affects each — lets you triage in minutes instead of arguing about brand names.

The four numbers

These numbers trade off. A device doing 100 000 IOPS at 4 KB each delivers 400 MB/s of throughput. Doubling IOPS at smaller block size doubles IOPS but halves throughput.

flowchart LR
  A[Application issues I/O] --> Q[Queue depth: how many concurrent requests?]
  Q --> D[Device: NVMe / SSD / HDD]
  D --> R[Result: IOPS, latency, throughput]

Synchronous vs asynchronous writes

• Asynchronous write: the application issues the write, the kernel acknowledges it, the data sits in the page cache, eventually flushes to disk. Fast. If the host loses power between acknowledgement and flush, the write is lost.
• Synchronous write: the application issues the write, the kernel hands it to the device, the device confirms durable storage before returning. Slower (often much slower), but crash-safe.

Databases use synchronous writes to guarantee durability. Filesystem fsync() triggers them. ZFS has its own equivalent (ZIL). Ceph RBD has its own equivalent (write journal).

Queue depth

The device’s parallelism. Most storage devices have an internal queue that allows them to process multiple requests concurrently. NVMe supports 65 535 commands per queue; SATA SSDs typically 32.

To maximise throughput, the application (or kernel) must keep enough requests in flight. A single-threaded application issuing one I/O at a time wastes the device’s parallelism.

The storage stack in Proxmox

flowchart TB
  App[Application fsync/write] --> FS[Guest filesystem]
  FS --> VT[VirtIO-SCSI]
  VT --> QE[QEMU]
  QE --> HL[Host block layer]
  HL --> ZFS[ZFS / LVM / RBD]
  ZFS --> PD[Physical disk]

Every layer can be a bottleneck. Performance issues are diagnosed by measuring at each layer:

• Guest: iostat -x inside the guest.
• Host: iostat -x on the host.
• Storage: ZFS zpool iostat, Ceph ceph osd perf.

The most common anti-patterns

Benchmarking with fio

fio is the standard Linux storage benchmark. The Proxmox host has it in the package repositories.

apt install -y fio && fio --name=randread --ioengine=libaio --direct=1 --bs=4k --rw=randread --numjobs=4 --runtime=30 --size=1G --filename=/tmp/fiotest --group_reporting

fio --name=syncwrite --ioengine=libaio --direct=1 --bs=4k --rw=randwrite --numjobs=1 --runtime=30 --size=1G --filename=/tmp/fiotest --sync=1 --group_reporting

Production considerations

Common mistakes

• “I have an SSD, so storage is fast.” SSDs differ by 100× in IOPS; cheap consumer SSDs are slow under sustained writes.
• “Sync writes only matter if I have a database.” All workloads that care about durability (most of them) need to think about sync semantics.
• “Higher queue depth is always better.” No — it depends on the device’s ability to parallelise. Saturating queue depth on a single HDD does not help.

Key takeaways

• IOPS, latency, throughput, queue depth. Four numbers. Know your workload’s profile.
• Sync writes are limited by the slowest durable path, often the SLOG.
• Benchmarks must reflect real workloads.

Knowledge check