IOPS and block size

Intermediate⏱ ~16 minfio

What you'll learn

Explain the relationship between IOPS and block size
Convert between IOPS and bandwidth
Identify the crossover point for a device
Compare figures at different block sizes correctly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An IOPS figure without its block size is uninterpretable, and quoting one against another taken at a different size is the most common way performance comparisons go wrong.

The relationship

bandwidth = IOPS × block size

4k  at 50,000 IOPS  =  195 MiB/s
64k at 12,000 IOPS  =  750 MiB/s
1M  at    900 IOPS  =  900 MiB/s
4M  at    250 IOPS  = 1000 MiB/s

The IOPS number falls as the block size rises and the bandwidth rises. Neither figure alone describes the device.

The two regimes

Regime	Limited by	Block sizes
IOPS-bound	per-operation overhead	small
Bandwidth-bound	media or link throughput	large

for bs in 4k 16k 64k 256k 1M 4M; do
  fio --name=bs$bs --ioengine=rbd --pool=rbd-vms --rbdname=bench \
      --rw=randread --bs=$bs --iodepth=32 --runtime=30 --time_based \
      --output-format=json | python3 -c '
import sys,json; d=json.load(sys.stdin)["jobs"][0]["read"]
print("%-5s %8d IOPS %9.1f MiB/s" % ("'"$bs"'", d["iops"], d["bw"]/1024))'
done

Typical result on NVMe-backed RBD:

4k      98420 IOPS     384.5 MiB/s
16k     71200 IOPS    1112.5 MiB/s
64k     28400 IOPS    1775.0 MiB/s
256k     8900 IOPS    2225.0 MiB/s
1M       2300 IOPS    2300.0 MiB/s
4M        580 IOPS    2320.0 MiB/s

The crossover is visible: below 64k the IOPS number moves little relative to the block size, meaning per-operation cost dominates; above it, bandwidth flattens, meaning the link or media is the limit.

Finding the crossover

The block size where bandwidth stops rising is where the workload transitions from overhead-bound to throughput-bound:

4k → 16k:   bandwidth ×2.9    (overhead-dominated)
64k → 256k: bandwidth ×1.25   (approaching the limit)
1M → 4M:    bandwidth ×1.01   (throughput-bound)

Knowing where this sits for your hardware determines whether increasing an application’s I/O size will help.

Comparing correctly

Comparison	Valid?
Two clusters, both 4k random read at qd32	yes
4k on one, 64k on the other	no
Same cluster before and after, same profile	yes
A vendor’s 4k figure against your 4M figure	no

"98,000 IOPS"                      ← uninterpretable
"98,420 IOPS 4k random read qd32"  ← comparable

Quiz

Knowledge check · 4 questions

Q1. Why does IOPS fall far less than proportionally as block size increases?

Larger blocks compress better, so the bytes reaching the device grow more slowly than the request size The device buffers large blocks in its own DRAM and acknowledges them before any of it reaches the media Larger blocks are striped across several OSDs in parallel, so one request is served by several devices at once Fixed per-operation costs — round trip, metadata lookup, transaction — dominate small operations and are amortised across large ones
Q2. A cluster measured at 2,300 IOPS can be moving six times more data per second than one measured at 98,000 IOPS.

True False
Q3. Evaluate a vendor performance claim.

A vendor claims 1,000,000 IOPS and 12 GB/s for a storage system. An application team wants to know whether it will meet their 4k random write requirement.
Q4. How do you find the block size at which a system transitions from overhead-bound to throughput-bound?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Quote IOPS with its block size, pattern, and queue depth or the figure cannot be compared with anything. Sweep block sizes to find where bandwidth flattens on your hardware — that crossover tells you whether increasing an application’s I/O size will help.

Cross-course references

Kubernetes: throughput claims need the same request-size qualification
Linux: per-operation overhead versus data volume is the same distinction everywhere