The layered model for performance work

Intermediate⏱ ~17 mincephfioiperf3

What you'll learn

Describe the layers a Ceph I/O passes through
Attribute latency to the correct layer
Choose the right tool for each layer
Avoid the common misattributions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“Ceph is slow” is not a diagnosis. Every I/O crosses five layers and the latency belongs to one of them; naming the layer is most of the work.

The layers

flowchart TD
  A[Application] --> B[Client library / kernel driver]
  B --> C[Network]
  C --> D[OSD daemon]
  D --> E[Device]
  E --> D
  D --> C
  C --> B
  B --> A

Layer	Contributes	Measured with
Application	queue depth, I/O size, sync behaviour	application metrics, `strace`
Client	librbd cache, kernel driver, mount options	`rbd perf image iostat`, client logs
Network	round trip, congestion, retransmits	`iperf3`, `ping`, interface counters
OSD	queueing, threading, scheduler	`ceph osd perf`, `ceph daemonperf`
Device	seek, service time, controller	`fio`, `iostat`, SMART

Attributing latency

The arithmetic that anchors an investigation:

client-observed latency
  = network RTT
  + OSD queue wait
  + OSD processing
  + device service time
  + replication fan-out (the slowest replica)

If the client sees 40 ms and the device reports 2 ms, 38 ms is above the device — and that is a very different investigation from the one where the device reports 35 ms.

# device
PEER=peer
iostat -x 1 5 | awk '/^nvme|^sd/ {print $1, $NF}'

# OSD
ceph osd perf | sort -k2 -n | tail -5

# network
iperf3 -c ${PEER} -t 10
ping -c 100 ${PEER} | tail -2

Common misattributions

Symptom	Assumed cause	Frequent actual cause
High client latency	slow disks	one slow OSD dragging every PG it holds
Slow writes	network	replication waiting on the slowest replica
Slow reads	Ceph	client queue depth of 1
Latency during recovery	recovery	a device exposed by recovery’s extra load
Latency on one client	the cluster	that client’s mount options or CPU

The last row is worth an explicit check early: comparing two clients against the same pool separates client-side from cluster-side in one step.

# from two different client hosts, same pool
fio --name=probe --ioengine=rbd --pool=rbd-vms --rbdname=probe \
    --rw=randread --bs=4k --iodepth=1 --runtime=60 --time_based

Working the layers in order

1. Is it one client or all clients?     → client layer or below
2. Is it one pool or all pools?         → pool config or below
3. Is it one OSD or all OSDs?           → device/host or cluster-wide
4. Does the device report the latency?  → device or above
5. Does the network show loss or delay? → network or OSD

Each question is answerable in a minute and each one halves the search space.

Quiz

Knowledge check · 4 questions

Q1. Why does a single slow OSD affect p99 latency disproportionately?

Every write to a PG in its acting set waits for it, so the PGs it holds show its latency while the median is unaffected A slow OSD is promoted to primary for its PGs, so it serves every read and write for them directly The monitors shift client traffic towards it while it is slow, to keep its queue draining CRUSH raises its weight as its queue grows, so it receives a larger share of newly written objects
Q2. If the client observes 40 ms and the device reports 2 ms service time, the disks are the problem.

True False
Q3. Begin a performance investigation.

A team reports that Ceph has become slow. They provide no further detail. Cluster health is HEALTH_OK.
Q4. What five questions narrow a performance investigation fastest?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Run the same test from two client hosts before investigating the cluster — a large share of “Ceph is slow” reports are one client’s queue depth or mount options. Compare client-observed latency against device-reported service time early; the difference tells you which side of the device the problem is on.

Cross-course references

Kubernetes: distinguishing pod, node, and cluster causes follows the same layering
Linux: the classic application-to-device latency decomposition is identical