The per-OSD benchmark

Advanced⏱ ~17 minceph

What you'll learn

Run the per-OSD benchmark
Interpret its result against the device class
Understand its role in mClock capacity measurement
Use it to identify hardware outliers

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

This benchmark tests one OSD’s device path directly, which makes it the cleanest way to compare OSDs against each other and against their class.

Running it

ceph tell osd.12 bench

{
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 5.42,
    "bytes_per_sec": 198106284,
    "iops": 47.2
}

# with explicit parameters: total bytes, block size, object size, object count
ceph tell osd.12 bench 12288000 4096 4194304 100

Parameter	Meaning
total bytes	how much to write
block size	the I/O size
object size	size of each object created
object count	number of objects

Small block sizes measure the random write path; the default 4 MiB measures throughput.

Comparing across OSDs

for osd in $(ceph osd ls); do
  printf '%-6s ' "osd.$osd"
  ceph tell osd.$osd bench 12288000 4096 4194304 100 2>/dev/null | \
    python3 -c 'import sys,json; d=json.load(sys.stdin); print(round(d["iops"],1), "IOPS")'
done | sort -k2 -n

This produces a ranked list of every OSD’s measured capability, which is the most direct way to find a hardware outlier. Devices of the same model in the same role should cluster tightly; one well below the others is a finding.

The mClock connection

ceph config get osd osd_mclock_max_capacity_iops_hdd
ceph config get osd osd_mclock_max_capacity_iops_ssd
ceph config show osd.12 | grep mclock_max_capacity

mClock derives its allocations from a measured per-OSD capacity, obtained by exactly this benchmark at OSD startup. If that measurement is taken while the device is busy or briefly slow, mClock’s allocation for that OSD is wrong for the life of the daemon:

# reset a bad measurement
ceph config rm osd.12 osd_mclock_max_capacity_iops_hdd
ceph orch daemon restart osd.12

A cluster where one OSD behaves oddly under mClock is worth checking for exactly this.

Cautions

Caution	Why
It writes to the OSD	it consumes capacity until cleaned up
It competes with client I/O	run during a quiet period where possible
Results vary with cluster state	compare runs taken under similar conditions
It measures one OSD’s path	not the cluster’s end-to-end behaviour

Quiz

Knowledge check · 4 questions

Q1. Why does mClock measure each OSD's capacity rather than using a configured value?

So the scheduler can detect a failing device and automatically reduce the work it dispatches to that OSD Because a configured value would have to be identical on every OSD, since the option is set cluster-wide Because a configured value is not persisted across a daemon restart, while a measured one is kept in the config store Its allocations are proportions of total capacity, which differs by an order of magnitude between device classes and varies within them
Q2. A capacity measurement taken while an OSD was briefly slow corrects itself over time.

True False
Q3. Investigate an OSD behaving oddly under mClock.

One OSD consistently contributes less recovery throughput than its identical peers and shows lower client throughput, though its device benchmarks normally when tested directly.
Q4. How does `ceph tell osd bench` help identify a hardware outlier?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Run ceph tell osd bench across every OSD and sort the results — identical devices should cluster tightly and an outlier is a direct finding. When an OSD underperforms under mClock but benchmarks normally, compare its stored osd_mclock_max_capacity_iops_* against its peers and force a re-measurement.

Cross-course references

Kubernetes: node capacity detection errors mis-schedule for the node’s lifetime
Linux: any auto-tuned parameter measured once at boot carries the same risk