fio and safe benchmarking - measuring storage baseline

fio (Flexible I/O Tester) is the standard tool for storage benchmarking. It measures latency, throughput, and IOPS for specific workload patterns. This lesson covers safe and useful benchmarking.

Install fio

sudo apt install fio
# or
sudo dnf install fio

A basic test

fio --name=randread \
    --filename=/mnt/test/fio-test.bin \
    --size=1G \
    --bs=4k \
    --rw=randread \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=32 \
    --runtime=30 \
    --time_based

Parameters:

• --name: name of the test.
• --filename: file to read/write. Use a separate file, not production data.
• --size: file size. 1G is a reasonable test size.
• --bs: block size. 4k for database, 64k for streaming.
• --rw: read/write pattern. randread, randwrite, read, write, randrw.
• --ioengine: libaio (kernel async) or psync (slower, simpler).
• --direct: 1 bypasses the page cache. It defaults to 0, so omitting it is the single most common way to produce a wrong baseline: you measure RAM, not storage. It also costs you the queue depth — fio documents that libaio “may only support queued behavior with non-buffered I/O”, so a buffered job runs synchronously and --iodepth=32 means nothing. Always pass it.
• --iodepth: queue depth. 32 is typical.
• --runtime: time-based test duration.
• --time_based: run for --runtime even if --size is written.

Output

  read: IOPS=12,345, BW=48.2MiB/s (50.5MB/s)(1475MiB/30001msec)
    slat (usec): min=2, max=125, avg=4.50
    clat (usec): min=15, max=8500, avg=78.50
     lat (usec): min=18, max=8505, avg=83.00
    bw (MiB/s): min=45, max=52, per=100.00%

Reading:

• IOPS: 12,345 read operations per second.
• BW: bandwidth, 48.2 MiB/s.
• clat: completion latency. avg=78.50 us, max=8500 us (8.5ms). High max = inconsistent performance.
• slat: submission latency. Submission to driver.
• lat: total latency.

For production baseline, capture IOPS, BW, and latency at p50, p95, p99, p99.9.

Safe benchmarking

fio can destroy data. To benchmark safely:

1. Use a separate test file (not production data).
2. Use a separate test directory (not /, not /var).
3. Never benchmark the root filesystem.
4. Check free space before you start. fio lays out the whole --size file up front, so a 10G test needs 10G free plus headroom.
5. Stop the test if it interferes with production.
6. Document the test parameters.

Never place a fio test file on a database data directory, a log directory, or the root filesystem. fio creates the file if it does not exist and fills it to --size. Ten gigabytes appearing in /var/lib/mysql fills the datadir, InnoDB can no longer extend the tablespace or write the redo log, and the server aborts writes. If binlogs live on the same filesystem, replication breaks with it. The recovery is a crash restart, not a rm.

For read tests, use a file that has been pre-filled:

# Create the test file
dd if=/dev/zero of=/mnt/test/fio-test.bin bs=1M count=1024

# Run read test (does not modify the file)
fio --name=randread --filename=/mnt/test/fio-test.bin ...

For write tests, use a separate test directory that you can wipe.

Common workload patterns

# Database: random 4K reads
fio --rw=randread --bs=4k --iodepth=32

# Database writes: random 4K writes
fio --rw=randwrite --bs=4k --iodepth=32

# Streaming read: sequential 1M
fio --rw=read --bs=1M --iodepth=4

# Streaming write: sequential 1M
fio --rw=write --bs=1M --iodepth=4

# Mixed: 70% read, 30% write
fio --rw=randrw --rwmixread=70 --bs=4k --iodepth=32

Each pattern tests a different aspect of storage performance.

Capture baseline

For a database host, benchmark the same storage class the database uses, but never inside the data directory itself. Use a scratch path you can wipe.

# 1. Scratch path: not the datadir, not /var, not /
sudo mkdir -p /mnt/fio-test

# 2. Confirm room. A 10G test needs 10G plus headroom.
df -h /mnt/fio-test

# 3. Confirm the scratch path is not on the datadir filesystem
findmnt -no SOURCE,TARGET -T /mnt/fio-test
findmnt -no SOURCE,TARGET -T /var/lib/mysql

Run the benchmark only if df reports at least 20G available and the two findmnt lines name different filesystems. If they match, stop and pick another path.

# Random 4K reads at queue depth 32
fio --name=baseline-randread \
    --filename=/mnt/fio-test/test.bin \
    --size=10G \
    --bs=4k \
    --rw=randread \
    --ioengine=libaio \
    --direct=1 \
    --iodepth=32 \
    --runtime=60 \
    --time_based \
    --output=baseline-randread.json \
    --output-format=json

Cleanup. fio does not remove the test file:

rm -f /mnt/fio-test/test.bin
df -h /mnt/fio-test

Save the JSON result. Compare future benchmarks against the baseline.

To characterise the device that holds the datadir, do it before the database is in service, or on an identical host that is out of rotation. A live datadir is never a benchmark target.

Interpret results

Key metrics:

• IOPS: operations per second.
• BW: throughput.
• clat p50, p99, p99.9: latency percentiles. p99.9 is the “tail” - rare slow operations.
• clat stddev: standard deviation. Low = consistent.

A healthy storage has:

• High IOPS for the workload.
• Low latency at all percentiles.
• Low standard deviation (consistent performance).

Storage with issues:

• High average IOPS but high p99.9 latency = inconsistent (queue contention, GC, etc.).
• Low IOPS = slow storage.

Knowledge check