Ceph performance and health investigation · Proxmox VE

Why this matters in production

“Ceph is slow” is a production ticket. The fix depends on where the slowness is. This lesson teaches the diagnostic flow.

The diagnostic flow

flowchart LR
  A[VM reports slow I/O] --> B[Where is latency?]
  B --> C[Client side]
  B --> D[Network]
  B --> E[OSD side]

Step 0: Rule out the four states that are not performance problems

Before measuring anything, establish that the cluster is not doing something that should make it slow. Each of these produces the same ticket and none of them is a tuning question.

Read-only / Safethe sixty-second triage— Read-only. If any of these four is true, stop: the cluster is behaving correctly for the state it is in, and the fix is to change the state rather than to tune anything.

# 1. Is a recovery or backfill running?
ceph -s | grep -E 'recovery|backfill|misplaced|degraded'

# 2. Is a deep scrub running right now?
ceph pg dump | awk '$0 ~ /scrubbing/ {print $1, $10}' | head

# 3. Is any OSD near full? Allocation slows sharply as OSDs fill.
ceph osd df | sort -k17 -n -r | head -5

# 4. Are any flags set that change behaviour?
ceph osd stat

A cluster recovering from a failed OSD is slow because it is doing the work it exists to do. A cluster whose OSDs are at 88% is slow because the allocator is working harder for every write. Both look identical from a guest, and neither is fixed by anything in the rest of this lesson.

Step 1: Confirm the symptom from the client

Service impact possiblemeasure inside the guest— fio writes and reads a real file. --direct=1 bypasses the guest page cache so you measure the storage path rather than RAM. Run it on a test guest, not on a production database - it generates sustained I/O and the file it creates occupies real space.

fio --name=randread --ioengine=libaio --direct=1 --bs=4k \
--rw=randread --numjobs=1 --runtime=10 --size=1G \
--filename=/tmp/fio --group_reporting

rm -f /tmp/fio

Note IOPS, latency, and throughput. If guest-side latency is fine, the issue is in the guest.

Step 2: Measure from the host

Service impact possiblemeasure the same thing from outside the guest— Measures the Ceph path without QEMU in it. Point --filename at a scratch RBD image you created for the purpose; running fio against a device holding a live VM disk will corrupt it.

# create a scratch image rather than testing against a VM disk
POOL=vm-storage
rbd create "$POOL/fio-scratch" --size 10G
DEV=$(rbd map "$POOL/fio-scratch")

fio --name=randread --ioengine=libaio --direct=1 --bs=4k \
--rw=randread --numjobs=1 --runtime=10 --size=1G \
--filename="$DEV" --group_reporting

rbd unmap "$DEV"
rbd rm "$POOL/fio-scratch"

If host-side latency is also fine but guest-side was not, the issue is in guest configuration (cache, scheduler, etc.).

Benchmark the cluster, not a disk, with `rados bench`

fio against an RBD image measures the whole path including the client. rados bench measures RADOS itself, which separates “Ceph is slow” from “this client is slow”.

POOL=vm-storage
rados bench -p "$POOL" 30 write --no-cleanup
rados bench -p "$POOL" 30 seq
rados -p "$POOL" cleanup

--no-cleanup on the write pass leaves the objects so the read passes have something to read. The cleanup at the end is not optional — those objects occupy real pool capacity until you remove them, and a forgotten benchmark is a genuine way to fill a small pool.

Step 3: Check Ceph health

ceph -s && ceph osd perf

ceph osd perf shows per-OSD latency and throughput. A spread of latencies across OSDs indicates a few hot OSDs; a uniformly elevated latency indicates a cluster-wide issue.

Step 4: Use OSD perf counters

ceph daemon osd.0 perf dump | head -100

Key counters:

Counter	Meaning
`op_latency_avg`	Average latency per OSD op
`op_latency_max`	Tail latency
`op_queue_latency_avg`	Time ops spend queued
`op_process_latency_avg`	Time spent processing

If op_queue_latency is high, OSDs are saturated. If op_process_latency is high, the disk is the bottleneck.

Step 5: Check network

Read-only / Safe— Check NIC counters for errors and utilisation.

for nic in $(ls /sys/class/net | grep -v lo); do echo "=== $nic ==="; ethtool -S $nic 2>/dev/null | grep -E 'rx_errors|tx_errors|dropped' | head -5; done

Common causes of “Ceph is slow”

Cause	Indicator	Fix
OSDs saturated	High op_queue_latency	Add OSDs, add hosts, throttle clients
Slow disks	High op_process_latency	Replace disks
Network saturation	High NIC utilisation; drop counter increasing	Add bandwidth; isolate networks
PG count too low	Few PGs, low recovery throughput	Increase PG count
Cache cold	First IO slow, subsequent fast	Expected behaviour
Backfill/recovery in progress	`ceph -s` shows active recovery	Wait or throttle

Latency breakdown

Ceph I/O latency is the sum of:

Client → librbd → messenger → OSD op queue → OSD op process → reply

Each step adds latency. To find the bottleneck, measure at each step.

Step	Typical latency
Client to messenger	0.1–1 ms
Messenger to OSD	0.1–1 ms
OSD op queue	0–5 ms (saturated)
OSD op process (write)	1–10 ms (depends on replication)
OSD op process (read)	0.5–5 ms

A replication=3 write has 3× the OSD op process latency (one per replica) plus network RTT between OSDs.

Under the hood

Why a size 3 write costs more than three times one write, and where the extra goes.

A client write to an RBD image does not fan out to three OSDs in parallel from the client. The client sends the write to the primary OSD for that placement group. The primary writes it locally and forwards it to the two replicas. The write is acknowledged to the client only when both replicas have confirmed.

So the client-visible latency is:

client -> primary            (one network hop)
primary -> replica, replica  (one more hop, in parallel)
replica commit               (the slowest of the two)
replica -> primary ack
primary -> client ack        (one more hop)

Three network traversals, plus the commit latency of the slowest replica. That last point is the one with operational consequences: a size 3 write is as slow as the worst of the three OSDs involved, so a single failing disk with rising latency degrades every placement group it participates in — not one third of them, all of them.

This is why ceph osd perf is the highest-value command in this lesson. It reports commit latency per OSD, and one OSD with an order-of-magnitude outlier explains a cluster-wide slowdown that no aggregate metric would attribute to it.

ceph osd perf | sort -k3 -n -r | head

Two further consequences follow from the same structure.

The cluster network carries roughly twice the client write volume. Each client write becomes two replica writes across it. Sizing the cluster network at the same speed as the public network is therefore under-provisioning it by design, which is why the sizing lesson asks for more.

Reads do not fan out. A read is served by the primary OSD alone, so read latency is one hop plus one disk, and read performance scales with OSD count rather than with replica count. size 3 costs write performance and buys nothing for reads — worth knowing when someone proposes raising size to make things faster.

What happens if one disk is failing slowly rather than failing outright?

The whole cluster gets slower and nothing reports a fault.

A disk that has failed is easy: the OSD goes down, PGs go degraded, recovery starts, and there is a health warning naming it. A disk that is dying — reallocating sectors, retrying reads, taking 400 ms to service an I/O that used to take 2 ms — stays up and stays in. Ceph keeps sending it its share of the work, and because a size 3 write waits for the slowest replica, every PG that OSD participates in slows to its speed.

The symptom reported to you is “everything is slow”, cluster-wide, with HEALTH_OK and no failed hardware.

What identifies it, in order:

ceph osd perf | sort -k3 -n -r | head
ceph health detail | grep -i slow

ceph osd perf is decisive: one OSD with a commit latency far above its siblings is the answer, and it takes seconds. ceph health detail may also report SLOW_OPS naming the OSD once the latency is bad enough to trip the threshold.

Confirm it at the hardware layer before acting:

smartctl -a /dev/sdX | grep -iE 'reallocated|pending|error|wear'
dmesg -T | grep -iE 'sd[a-z]|nvme|I/O error|medium error'
journalctl -u "ceph-osd@5" --since '2 hours ago' | tail -50

Then treat it as a replacement rather than a performance problem: ceph osd out, wait for the rebalance, and follow the day-2 replacement procedure. Marking it out immediately removes it from the write path, which restores cluster performance before the disk is physically touched — so the fix and the fault diagnosis are the same action.

The reason this is worth rehearsing is that the instinct it triggers is wrong. “Everything is slow” sends people to network counters, to the recovery throttles, to PG counts — all cluster-wide explanations for a cluster-wide symptom. The cause is one device, and the command that finds it is not in that search space.

Production considerations

Common mistakes

Optimising the wrong layer. Always measure from the symptom outward.
Tuning before ruling out recovery, deep scrub, nearfull OSDs and set flags. A cluster doing correct work slowly is not a tuning problem.
Looking for a cluster-wide cause when one OSD is dying slowly. ceph osd perf finds it in seconds and no aggregate metric will.
Raising size to improve performance. Reads are served by the primary alone, so more replicas cost write latency and buy nothing for reads.
Running fio against a device that holds a live VM disk. Create a scratch RBD image.
Leaving rados bench objects behind. They occupy real pool capacity.
Trusting ceph -s HEALTH_OK without checking performance counters.
Believing latency is “fine” because the average is low. Tail latency is what users feel.
Adding capacity when the bottleneck is throughput, not capacity.

Key takeaways

Rule out recovery, scrub, nearfull and flags before measuring anything.
Measure from the guest inward: fio in the guest, fio on a scratch RBD image, then rados bench for RADOS itself.
A size 3 write waits for the slowest of three OSDs, so one degraded disk slows every PG it touches. ceph osd perf is how you find it.
ceph osd perf and ceph daemon osd.X perf dump give OSD-level detail.
Tail latency matters; monitor p95 and p99.