Lab: Storage baseline with fio

This lab runs fio benchmarks for multiple workload patterns and produces a storage baseline. By the end you will have the discipline for storage performance verification.

Tasks

Task 1: Set up the test environment

Benchmark the device you mean to benchmark, on a host you can lose

/mnt/fio-test is a directory on whatever filesystem /mnt lives on, which on a default install is /. Two consequences, and the second is the one that ends the exercise badly.

The numbers are wrong: you measure the root filesystem — its mount options, its scheduler, whatever else is writing to it — and label the result as the storage device’s baseline. If the device under test is not the one holding /, the benchmark never touched it.

The host may run out of space: the pre-fill below writes 4 GiB, the random-write tests write more, and dd on a filesystem with less than that free fills it. At 100 per cent on /, journald stops recording, PAM cannot write wtmp and sshd cannot create new sessions — on a remote host with no console that is a rescue boot.

Run this on a disposable host, mount the device under test explicitly, and check the free space first.

# Mount the device under test - not a directory on the root filesystem.
# Substitute the device you actually intend to benchmark.
DEV=/dev/nvme1n1
sudo mkdir -p /mnt/fio-test
sudo mount "$DEV" /mnt/fio-test
findmnt /mnt/fio-test          # confirm it is a separate mount before continuing

# The test file needs one owner, not world-write. 0777 on a directory lets
# any local user replace the file mid-run and silently change your results.
sudo install -d -o "$USER" -g "$USER" -m 0755 /mnt/fio-test

# Confirm there is room for the pre-fill plus headroom, then write it
df -h /mnt/fio-test
fio --name=prefill \
    --filename=/mnt/fio-test/test.bin \
    --size=4G \
    --bs=1M \
    --rw=write \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=8 \
    --refill_buffers

The pre-fill uses fio rather than dd if=/dev/zero on purpose. An all-zero file is the most compressible data there is, so a device or layer that compresses or deduplicates — many SSD controllers, ZFS, a thin-provisioned SAN LUN — stores almost nothing and returns the reads from nowhere. The result is a baseline for a file that does not physically exist. --refill_buffers makes fio write fresh incompressible data for every buffer.

If no spare device is available, a loop-backed file on a filesystem you can afford to fill is a valid substitute for learning the tool — but say so in the report. A loop device measures the backing filesystem, not the hardware, so the figures are not a storage baseline.

Task 2: Random read test (database)

Every job below carries --direct=1, and the baseline is worthless without it

direct defaults to false in fio, which means buffered I/O. Without --direct=1 the reads are served from the page cache whenever the working set fits in RAM, so the figure you file as the array’s IOPS is really a measurement of memory bandwidth — and it will be spectacular, which is why the mistake survives review.

It also silently disables the queue depth you asked for. fio’s own documentation says of libaio: “Linux may only support queued behavior with non-buffered I/O (set direct=1 or buffered=0).” Buffered, --iodepth=32 degrades to synchronous one-at-a-time I/O, so the job neither measures the device nor exercises the concurrency it claims to.

If a baseline in your runbook has no --direct=1 in the recorded command line, it is not a baseline. Record the full command line alongside the numbers so this is checkable later.

fio --name=randread \
    --filename=/mnt/fio-test/test.bin \
    --size=2G \
    --bs=4k \
    --rw=randread \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=32 \
    --runtime=30 \
    --time_based \
    --output=/tmp/fio-randread.json \
    --output-format=json

Task 3: Random write test (database)

fio --name=randwrite \
    --filename=/mnt/fio-test/test.bin \
    --size=2G \
    --bs=4k \
    --rw=randwrite \
    --ioengine=libaio \
    --direct=1 \
    --refill_buffers \
    --iodepth=32 \
    --runtime=30 \
    --time_based \
    --output=/tmp/fio-randwrite.json \
    --output-format=json

Task 4: Sequential read test (streaming)

fio --name=seqread \
    --filename=/mnt/fio-test/test.bin \
    --size=2G \
    --bs=1M \
    --rw=read \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=4 \
    --runtime=30 \
    --time_based \
    --output=/tmp/fio-seqread.json \
    --output-format=json

Task 5: Mixed read/write test (typical app)

fio --name=randrw \
    --filename=/mnt/fio-test/test.bin \
    --size=2G \
    --bs=4k \
    --rw=randrw \
    --rwmixread=70 \
    --ioengine=libaio \
    --direct=1 \
    --refill_buffers \
    --iodepth=32 \
    --runtime=30 \
    --time_based \
    --output=/tmp/fio-randrw.json \
    --output-format=json

Task 6: Extract key metrics

For each test, extract IOPS, throughput, p99 latency:

# Use jq to extract
jq '.jobs[0].read.iops, .jobs[0].read.bw, .jobs[0].read.clat_ns.percentile["99.000000"]' /tmp/fio-randread.json

Task 7: Document the baseline

STORAGE BASELINE
================
Host: <host>
Date: 2026-08-09
Device: /dev/sdb, SATA SSD, mounted at /mnt/fio-test
Command line: recorded verbatim with every job (must contain --direct=1)

Random read (4K, queue 32):
- IOPS: 12345
- Throughput: 48.2 MB/s
- Latency p50: 2.5 ms
- Latency p99: 8.5 ms

Random write (4K, queue 32):
- IOPS: 9876
- Throughput: 38.6 MB/s
- Latency p50: 3.2 ms
- Latency p99: 10.1 ms

Sequential read (1M, queue 4):
- Throughput: 520 MB/s
- Latency p50: 7.6 ms
- Latency p99: 14.2 ms

Mixed read/write (4K, 70% read, queue 32):
- Read IOPS: 8500
- Write IOPS: 3700
- Latency p50: 3.8 ms
- Latency p99: 12.0 ms

Sanity-check the numbers against the class of device before you file them. Those figures are a SATA SSD: 12,345 IOPS sits at the bottom of the 10,000-100,000 band, and the 2.5 ms p50 follows from the queue depth (32 in flight / 12,345 per second is about 2.6 ms each). An NVMe device in the same test should land in the hundreds of thousands of IOPS with a p50 in the tens to low hundreds of microseconds — see linux-io-latency-throughput-iops.

A baseline labelled with the wrong device class is worse than none. Milliseconds recorded as “NVMe” become the number an incident compares against, and the array will look healthy while it is an order of magnitude down.

Cleanup

The lab is not finished until the space is back. test.bin is 4 GiB and the fio runs add more.

# Remove the test file, then the mount
rm -f /mnt/fio-test/test.bin
sudo umount /mnt/fio-test
sudo rmdir /mnt/fio-test

# The JSON output stays - it IS the baseline. Move it somewhere durable.
mkdir -p ~/baselines
mv /tmp/fio-*.json ~/baselines/

# Confirm the host is back to its starting state
findmnt /mnt/fio-test          # no output
df -h /                        # free space unchanged from before the lab

If umount reports “target is busy”, find the holder with sudo lsof +D /mnt/fio-test and stop it. Do not use umount -l: a lazy unmount hides the mount but leaves the device attached and the space still consumed.

Objectives

Prerequisites