The device latency floor

Why this matters in production

A request for 1 ms write latency on an HDD-backed pool cannot be met by any configuration change. Knowing the floor turns that from an investigation into a conversation about hardware.

The floors

HDD latency is mechanical: a seek plus rotational delay. No firmware, driver, or Ceph setting reduces it.

From device floor to cluster latency

client write latency ≥ network RTT
                     + OSD processing
                     + slowest replica's device write

For a size=3 pool on HDD with a 0.2 ms network RTT:

≈ 0.2 ms (network)
+ 0.5 ms (OSD processing)
+ 10  ms (slowest of three HDD writes)
≈ 11 ms, before any queueing

The device term dominates so completely that everything else is noise — which is precisely why HDD clusters are configured with NVMe for the DB and WAL.

What the DB device changes

ceph osd metadata 12 | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("data:", d.get("bluestore_bdev_type"), d.get("bluestore_bdev_dev_node"))
print("db:  ", d.get("bluefs_db_type"), d.get("bluefs_db_dev_node"))'

BlueStore’s metadata operations — the RocksDB writes that record where data lives — go to the DB device. On an HDD OSD with an NVMe DB, those operations run at NVMe latency while the data writes run at HDD latency. For metadata-heavy workloads the improvement is substantial; for large-object streaming it is marginal.

Setting expectations

# measure your actual floor rather than assuming the table
fio --name=floor --filename=/dev/sdX --direct=1 --rw=randread \
    --bs=4k --iodepth=1 --runtime=30 --time_based --readonly

Identifying a device below its class

A device measuring well above its class range is failing or misconfigured:

smartctl -a /dev/sdX | grep -iE 'pending|reallocated|error'
smartctl -a /dev/sdX | grep -i 'rotation rate'   # confirm it is what you think
nvme smart-log /dev/nvme0n1 | grep -iE 'percentage_used|media_errors'
hdparm -W /dev/sdX                                # write cache state

A consumer SSD in a cluster is a common finding: it meets its class latency briefly and then collapses when its SLC cache is exhausted.

Quiz

Production discipline

Measure your devices’ actual floor with a queue-depth-1 test rather than assuming published figures, and run long enough to exhaust any SLC cache. When a latency requirement is below the device floor, present the decomposition — it converts an open-ended tuning request into a hardware decision.

Cross-course references

• Kubernetes: node hardware sets a floor no scheduling policy can beat
• Linux: fsync latency without write cache protection is the same order-of-magnitude effect