Disk benchmarking with fio

Before you can promise SLAs to applications, you need to know what your storage can actually deliver. This lesson shows how to use fio (Flexible I/O Tester) to get reproducible numbers from any Proxmox storage backend.

Installing fio

apt update && apt install -y fio

That installs the latest fio in the Debian repo. The upstream version is also fine to install manually from https://github.com/axboe/fio/releases if you need newer features.

The minimum useful benchmark

For a single SSD or disk, this gives you everything you need to know:

fio --name=baseline \
    --filename=/var/lib/vz/test.bin \
    --size=4G \
    --bs=4k \
    --rw=randread \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=32 \
    --runtime=30 \
    --time_based \
    --refill_buffers \
    --norandommap \
    --randrepeat=0 \
    --group_reporting \
    --output-format=normal

What each flag does:

The output tells you:

• IOPS — how many reads per second the disk delivers.
• lat (usec) — average latency in microseconds.
• clat percentiles — p50, p99, p99.9 in microseconds. Always read p99, not the average. Average latency hides tail-latency spikes that ruin application responsiveness.

The four benchmarks you actually need

Run all four on every new storage backend. Save the output in a versioned directory — you’ll compare to these baselines forever.

1. Random read (most common workload)

fio --name=randread \
    --filename=/var/lib/vz/test.bin \
    --size=4G --bs=4k --rw=randread \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json | tee /var/log/fio/randread.json

Expected on a modern SATA SSD: 50k–100k IOPS, p99 latency under 500 µs. NVMe: 200k–1M IOPS, p99 under 200 µs.

2. Random write (databases, mail servers)

fio --name=randwrite \
    --filename=/var/lib/vz/test.bin \
    --size=4G --bs=4k --rw=randwrite \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json | tee /var/log/fio/randwrite.json

Watch out: random writes to an SSD with no capacitor / power loss protection can be 10x slower than reads. Enterprise SSDs (with PLP) are rated for sustained random write at near-rating speed.

3. Sequential read (backups, restores)

fio --name=seqread \
    --filename=/var/lib/vz/test.bin \
    --size=4G --bs=1M --rw=read \
    --ioengine=libaio --direct=1 --iodepth=4 \
    --runtime=30 --time_based --norandommap \
    --output-format=json | tee /var/log/fio/seqread.json

Sequential read tells you how fast backups can be restored or how fast a VM can boot off this storage. NVMe: 3–7 GB/s. SATA SSD: 400–550 MB/s. Spinning disk: 100–200 MB/s.

4. Mixed 70/30 read/write (realistic workload)

fio --name=mixed \
    --filename=/var/lib/vz/test.bin \
    --size=4G --bs=4k --rw=randrw --rwmixread=70 \
    --ioengine=libaio --direct=1 --iodepth=32 \
    --runtime=30 --time_based --refill_buffers --norandommap \
    --output-format=json | tee /var/log/fio/mixed.json

This is closer to what a typical VM does: 70% reads, 30% writes. If your random read benchmark was 80k IOPS but mixed is only 20k, your disk is write-throttled (likely missing PLP).

Interpreting the numbers

The first thing to check on any storage backend is p99 latency:

Then look at IOPS at saturation. If your benchmark IOPS is the same with --iodepth=1 as --iodepth=32, your disk controller is the bottleneck, not the disk. If it scales to 32 but not 64, the queue depth is too low for the workload.

Finally, compare across disks in the same pool. A ZFS mirror should give you read IOPS equal to the sum of both disks. If it doesn’t, one disk is much slower (often a SATA SSD vs a faster NVMe mixed by mistake).

Benchmarking Ceph, NFS, and iSCSI

For network-attached storage, the test file should live on the backend, not on the local disk:

# For Ceph RBD
fio --name=ceph-randread \
    --ioengine=rbd \
    --pool=rbd \
    --rbdname=vm-100-disk-0 \
    --bs=4k --rw=randread \
    --iodepth=32 --direct=1 \
    --runtime=30 --time_based \
    --output-format=json | tee /var/log/fio/ceph.json

# For NFS
fio --name=nfs-randread \
    --directory=/mnt/pve/nfs-storage/ \
    --size=4G --bs=4k --rw=randread \
    --iodepth=32 --direct=1 \
    --runtime=30 --time_based \
    --output-format=json | tee /var/log/fio/nfs.json

# For iSCSI
fio --name=iscsi-randread \
    --filename=/dev/sdb \
    --bs=4k --rw=randread \
    --iodepth=32 --direct=1 \
    --runtime=30 --time_based \
    --output-format=json | tee /var/log/fio/iscsi.json

For network storage, add --runtime=60 because the first few seconds are often warm-up of TCP / RDMA connections. Watch the CPU steal time inside the PVE VM during the test — if it’s high, the network or storage backend is the bottleneck, not the VM.

Avoiding common pitfalls

• Don’t test on a full disk. fio’s test file needs free space. Use --size=4G on a storage backend with at least 10 GB free.
• Don’t run the test from inside a running VM. Test from the host, not the guest. Guest adds a layer of I/O scheduling that distorts the numbers.
• Don’t benchmark with the page cache hot. Always use --direct=1 and --refill_buffers to force disk I/O.
• Don’t trust a single run. Run each benchmark 3 times and take the median. The first run is often slower (cache warm-up, device readahead).
• Record the test conditions. Date, kernel, disk firmware, ZFS version. A “slow” benchmark six months from now is meaningless without the baseline conditions.

Saving baselines

Put the benchmarks in version control so future-you can compare:

mkdir -p /srv/fio-baselines/$(date +%Y-%m-%d)/
cp /var/log/fio/*.json /srv/fio-baselines/$(date +%Y-%m-%d)/
git -C /srv/fio-baselines add $(date +%Y-%m-%d)
git -C /srv/fio-baselines commit -m "fio baseline $(date +%Y-%m-%d)"

Whenever storage feels slow, re-run the same benchmarks and diff against the baseline. A 20% drop in IOPS or a 2x increase in p99 latency is the early warning that a disk is dying.

Production considerations

• Never run benchmarks on production. The test writes a 4 GB file that competes with real workload. Run during a maintenance window or on a test pool.
• Trust percentiles, not averages. A p99 of 50 ms with an average of 1 ms means 1% of your I/O is taking 50 ms — your database log flushes are stalling. Look at p99.9 and p99.99 for tail latency.
• Match block size to your real workload. Database workloads are 8 KB, mail servers are 16 KB, backup streams are 1 MB. One benchmark per workload, not one benchmark per disk.
• Validate RAID write performance. A mirror with a dead BBU can write 10x slower than rated. A short random-write benchmark is the cheapest way to catch this.

Common mistakes

• Benchmarking only sequential — sequential read on SSD is great; random write under load is what matters.
• Trusting vendor specs — a “550 MB/s” SATA SSD delivers that on sequential read only. Random 4K write is 30–80 MB/s.
• Skipping the sync / fsync test — for database workloads, run fio --rw=write --sync=1 to measure fsync latency. The number that matters is fsync p99, not write IOPS.
• Running fio from inside a VM — adds guest I/O scheduling and virtio overhead. The host is the right place.

Key takeaways

• Always use --direct=1 and --refill_buffers. Always read p99.
• Run four benchmarks: random read, random write, sequential read, mixed 70/30.
• Save baselines in version control. Re-run on every storage change.
• Network storage needs --runtime=60 for warm-up.

Knowledge check