rados bench · Ceph · RunBook Academy

rados bench

Intermediate⏱ ~17 minradosceph

What you'll learn

Run rados bench for write, sequential read, and random read
Interpret the output fields
Choose parameters that answer the question
Clean up afterwards

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

rados bench measures the object layer without RBD, CephFS, or RGW in the path. That makes it the right tool for answering “what can the cluster itself do”.

Running it

ceph osd pool create bench 128 128
ceph osd pool application enable bench rados

rados bench -p bench 60 write --no-cleanup
rados bench -p bench 60 seq
rados bench -p bench 60 rand
rados -p bench cleanup

The order matters: seq and rand read the objects that write --no-cleanup left behind. Running a read benchmark without a preceding write leaves nothing to read.

# useful parameters
rados bench -p bench 60 write -b 4096 -t 32 --no-cleanup

Flag	Meaning
`-b`	object size (default 4 MiB)
`-t`	concurrent operations (default 16)
`--no-cleanup`	keep the objects for read tests
`--run-name`	label, for parallel runs from several clients

Reading the output

Total time run:         60.041
Total writes made:      14231
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     948.16
Stddev Bandwidth:       112.4
Max bandwidth (MB/sec): 1104
Min bandwidth (MB/sec): 612
Average IOPS:           237
Stddev IOPS:            28
Average Latency(s):     0.0674
Stddev Latency(s):      0.0231
Max latency(s):         0.891
Min latency(s):         0.0192

Field	What it tells you
Bandwidth	sustained throughput at this object size
Stddev Bandwidth	consistency; high values suggest interference
Min bandwidth	worst-case sustained rate
Average Latency	mean per-operation latency
Max latency	the tail — often the most informative single number

A max latency far above the average points at an outlier OSD, a scrub, or contention rather than at the cluster’s general capability.

Choosing parameters

Question	Parameters
Maximum throughput	default 4 MiB objects, high `-t`
Small-object performance	`-b 4096`, high `-t`
Latency floor	`-b 4096 -t 1`
Sustained behaviour	a long duration, 300 s or more

Short runs measure cache and burst behaviour. A 60-second run on a large cluster may not even fill the caches, so a 300-second run is more representative for anything that will run continuously.

Cleaning up

rados -p bench cleanup
rados -p bench ls | wc -l          # should be 0
ceph osd pool delete bench bench --yes-i-really-really-mean-it

Leaving benchmark objects behind consumes capacity indefinitely and distorts later capacity figures. The --run-name used during the benchmark must match for cleanup to find the objects.

Quiz

Knowledge check · 4 questions

Q1. Why do rados bench sequential and random reads produce similar results?

Objects are placed pseudo-randomly by CRUSH, so reading in sequence still contacts effectively random OSDs The seq and rand modes issue the same request stream, and the flag only labels the output The client caches every object written during the preceding write phase, so both read modes are served locally The OSDs convert sequential object reads into random device reads to spread load across their PGs
Q2. `rados bench ... write --no-cleanup` removes its objects when the benchmark finishes.

True False
Q3. Benchmark a new cluster at the RADOS layer.

A cluster is being validated before production. The team wants to know its raw object-layer capability for both large and small objects.
Q4. What does a max latency far above the average indicate in rados bench output?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Run rados cleanup after any --no-cleanup benchmark and verify the object count is zero; the objects persist and a long high-throughput run leaves hundreds of gigabytes. Read the max latency alongside the average — a large gap points at an outlier or contention rather than at capability.

Cross-course references

Kubernetes: benchmark workloads left running consume the same silent capacity
Linux: any benchmark that creates files needs an explicit cleanup step