IOPS, throughput, latency — and why two of them fight

Why this matters in production

Every storage complaint arrives as “it is slow”, and the first job is to work out which of three different things the person means. They are related by simple arithmetic and they behave completely differently under load, so guessing wrong sends you tuning the wrong thing for a day.

• IOPS — operations per second. How many discrete requests the device or cluster completes.
• Throughput — bytes per second. How much data moves.
• Latency — time per operation. How long one request takes.

The arithmetic that binds them

For a workload with a fixed block size:

throughput = IOPS x block size

A device doing 10,000 IOPS at 4 KiB moves about 40 MiB/s. The same device doing 1,000 IOPS at 1 MiB moves about 1 GiB/s. Both numbers describe the same hardware; the workload chose which one looks good.

Latency relates to the others through concurrency, which is where intuition usually breaks:

IOPS ≈ queue depth / latency

At queue depth 1, a device with 1 ms latency does 1,000 IOPS and nothing you do to the device changes that. Raising queue depth to 32 can take it to 30,000 IOPS — with latency now considerably worse than 1 ms per operation, because requests are waiting in a queue. Nothing got faster. More work is in flight.

Which number does the workload care about?

The two rows that matter most for a Ceph operator are the last two. Recovery is a throughput-shaped workload sharing hardware with latency-shaped client workloads, and that collision is the single most common cause of “the cluster went slow” tickets.

Where each number is visible in Ceph

ID=12
POOL=rbd-vms
ceph -s                     # cluster-wide client IOPS and throughput
ceph osd perf               # per-OSD commit and apply latency, ms
ceph daemonperf osd.${ID}    # live per-OSD operation counters
rados bench -p ${POOL} 60 write   # deliberate throughput measurement

ceph -s reports what clients are getting. ceph osd perf is the one to reach for when latency is the complaint: it lists every OSD with its commit latency, and a single outlier in a column of otherwise similar numbers is a failing device far more often than it is a tuning problem.

Reading a complaint correctly

“The database is slow” almost always means latency. “The backup does not finish in the window” almost always means throughput. “The VMs hang when everyone logs in at 09:00” means IOPS under concurrency. Ask which one before touching anything, because the remedies are different and some of them are opposites: adding queue depth improves throughput and worsens latency, and throttling recovery improves client latency while making recovery take longer.

Quiz

Production discipline

Establish which of the three numbers a complaint is about before changing anything, and record all four values — IOPS, throughput, latency percentiles, and queue depth — whenever you measure. Watch distributions rather than means: ceph osd perf is the highest-value single command for latency complaints because one outlier device raises p99 for every placement group it participates in. And treat recovery as what it is — a throughput workload sharing devices with latency-sensitive clients.

Cross-course references

• Linux: Part XLI (Disk Performance) and Part XXXVIII (Performance Fundamentals) for the single-host view.
• Observability: histogram and percentile metrics are the right shape for all three numbers.
• Proxmox: Part VIII (Ceph) for the hyperconverged case where client and cluster workloads share hardware.