The performance impact of device class — the biggest lever you have

Why this matters in production

No tuning parameter in Ceph changes latency by two orders of magnitude. Device class does, which makes it the first question to answer about any pool and the last thing worth optimising around.

The scale of the difference

                random write commit    sequential throughput
HDD                    ~10 ms                150-250 MB/s
SATA SSD               ~0.3 ms              ~500 MB/s
NVMe                   ~0.05 ms             2000-7000 MB/s

Between HDD and NVMe that is roughly 200× on latency. Everything else an operator tunes — recovery throttles, memory targets, PG counts — moves latency by tens of percent.

Measuring per class

# per-OSD commit and apply latency, grouped by class
for o in $(ceph osd ls); do
  cls=$(ceph osd crush get-device-class osd.$o)
  lat=$(ceph osd perf -f json | jq ".osdstats.osd_perf_infos[] | select(.id==$o) | .perf_stats.commit_latency_ms")
  echo "$cls $lat"
done | awk '{s[$1]+=$2; n[$1]++} END {for (c in s) printf "%s mean commit %.2f ms over %d osds\n", c, s[c]/n[c], n[c]}'

And per pool, directly:

# Substitute your own values before running:
POOL=rbd-vms

rados -p "$POOL" bench 30 write -b 4096 -t 16 --no-cleanup
rados -p "$POOL" bench 30 rand
rados -p "$POOL" cleanup

Justifying with data

Before a purchase, measure what is actually slow:

# Substitute your own values before running:
OSD_ID=12                                        # an OSD from the list below
POOL=rbd-vms

ceph osd perf | sort -k2 -n | tail -10           # slowest OSDs
ceph daemon "osd.$OSD_ID" dump_ops_in_flight     # what they wait on
ceph df detail                                   # which pools are large
rados -p "$POOL" bench 30 write -b 4096 -t 16    # per-pool reality

A proposal that names the pool, the operation, the current latency, and the expected latency after the change is one that can be evaluated. A proposal for “faster storage” is not.

The decision, stated plainly

For each pool: which class, and why. Metadata and index pools on the fastest available. Latency-sensitive data on flash. Bulk and archive on HDD with BlueStore metadata on flash. Then measure and confirm.

Quiz

Production discipline

Answer the class question for every pool before tuning anything else, because no parameter in Ceph moves latency the way device class does. Spend scarce flash in order — BlueStore DB and WAL, then metadata pools, then latency-sensitive data, then bulk capacity — since ten devices at the first two points routinely beat sixty at the last. And measure before and after each change with rados bench and ceph osd perf, so the next budget conversation rests on evidence rather than assertion.

Cross-course references

• Ceph: Part LXVII (Performance Methodology) for the measurement discipline.
• Ceph: Part XII (BlueStore) for the DB and WAL placement.
• Ceph: Part LXXII (Benchmarking) for rados bench in depth.