Where Ceph is the wrong answer

Intermediate⏱ ~18 mincephfio

What you'll learn

Explain the architectural floor on write latency
Recognise metadata-bound filesystem workloads
Identify deployments too small to justify the overhead
Apply an honest test of operational capacity

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Deploying Ceph for a workload it cannot serve produces a cluster that is blamed for a decision made before it was installed.

The latency floor

Storage	Typical 4 KiB synchronous write
Local NVMe, direct	20 to 80 microseconds
Local NVMe behind a filesystem journal	100 to 200 microseconds
Ceph RBD on NVMe OSDs, 25 GbE	400 to 800 microseconds
Ceph RBD on SAS SSD OSDs	1 to 2 milliseconds
Ceph RBD on HDD with NVMe DB	5 to 20 milliseconds

fio --name=floor --filename=/dev/rbd0 --direct=1 --ioengine=libaio \
  --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --runtime=60 --time_based

Faster devices remove the device term. They do not remove the two
network traversals or the OSD code path, which is why the floor is
architectural rather than a hardware question.

Metadata-bound filesystem workloads

ceph fs status
ceph fs perf stats

Workload	Bottleneck
Millions of small files with heavy stat traffic	MDS, not OSDs
Deep directory trees walked repeatedly	MDS cache
Rename-heavy build trees	MDS, and renames across subtrees are expensive
Large sequential files	OSDs, which is the case Ceph serves well

# more metadata capacity is available, and it is real work
ceph fs set cephfs max_mds 2
setfattr -n ceph.dir.pin -v 0 /mnt/cephfs/tenants/a
setfattr -n ceph.dir.pin -v 1 /mnt/cephfs/tenants/b
ceph fs status

Multiple active MDS daemons with explicit subtree pinning solves this,
and it is an ongoing tuning commitment rather than a setting.

Too small to amortise

Deployment	Verdict
Under about 50 TiB usable, static	overhead dominates; use simpler storage
Three hosts, no growth planned, no failure domain requirement	hard to justify
A single application needing one filesystem	a NAS is cheaper and simpler
One team needing S3 for artefacts	a managed object service is cheaper

The team test

Can someone on call, at three in the morning, say what to do about a
placement group reported inconsistent, and how to tell whether it is
safe to run a repair?

ceph -s
ceph health detail
ceph osd pool ls detail

Answer	Implication
Yes, and it is written down	the capacity exists
Yes, one person knows	a single point of failure that takes holidays
No, we would open a support case	acceptable only with a support contract that answers at three in the morning
No	the deployment will drift into an unmaintained state

Why the write path has a latency floor that faster devices cannot remove

A replicated write travels from the client to the primary OSD, is written to the primary and forwarded to each replica in parallel, and is acknowledged only once every replica has committed. That is two network traversals in sequence plus the slowest replica commit plus the OSD processing on each hop. NVMe reduces the commit term to tens of microseconds and leaves the rest untouched, so the floor settles around three to five hundred microseconds on a good fabric. A workload requiring synchronous writes under a hundred microseconds — a database commit path, a low-latency message log — cannot be served by any distributed replicated store, and the correct answer is local storage with replication handled by the application.

Quiz

Knowledge check · 4 questions

Q1. Why does faster hardware not bring Ceph write latency close to local NVMe?

The CRUSH computation is expensive, and it runs once per operation on the client before the write can be issued The monitors serialise every write, so commit rate is bounded by the rate at which Paxos proposals can be committed The cluster network is always the bottleneck, so write latency is fixed by the fabric regardless of what devices the OSDs are built from The write crosses the network twice in sequence and waits for every replica to commit, and faster devices only shrink the commit term
Q2. A workload of millions of small files with heavy stat traffic is an OSD capacity problem on CephFS.

True False
Q3. Advise on a proposed Ceph deployment.

A team of two proposes a three-host cluster for 30 TiB of static data serving one application that commits synchronously at queue depth one. There is no support contract.
Q4. What does an under-operated Ceph cluster look like before it fails?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Compare the workloads tolerated latency against the architectural floor, not against a benchmark headline — two network traversals and a replica commit are not removable by hardware. Ask explicitly who answers a health warning at three in the morning and what they would do; a negative answer is a reason to choose something else, not a gap to fill later.

Cross-course references

Kubernetes: a platform nobody is funded to operate degrades silently for years
Linux: distributed replication buys durability by spending latency, always