High throughput is not low latency

Why this matters in production

Ceph runs two kinds of work on the same devices. Client I/O is latency-shaped: small, synchronous, and judged by how long each operation takes. Recovery, backfill, and scrub are throughput-shaped: large, asynchronous, and judged by how quickly they finish. They compete for the same queues, and the operator decides the split.

The mistake that follows is predictable. An OSD fails, recovery starts, someone wants the cluster back to active+clean quickly, and they raise the recovery throttles. Recovery does finish sooner. The VMs served by that cluster spend the whole period at ten times their normal latency, and the incident report records a storage outage that was actually a tuning decision.

The throughput-shaped activities

All four read and write large amounts as fast as they are permitted to. None of them is urgent in the way a client write is urgent — except recovery, which is restoring durability and therefore does have a deadline, just not a per-operation one.

The throttles

# how many PGs one OSD will backfill concurrently
ceph config get osd osd_max_backfills

# how many recovery operations per OSD at once
ceph config get osd osd_recovery_max_active

# a deliberate pause between recovery operations, in seconds
ceph config get osd osd_recovery_sleep

# scrub scheduling window
ceph config get osd osd_scrub_begin_hour
ceph config get osd osd_scrub_end_hour

These are runtime-adjustable, which is the point: they are meant to be changed while watching the effect, not set once at build time.

# during an incident: give clients more headroom
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_sleep 0.1

Which side to choose

The decision is about durability risk against service quality, and it has a clear structure:

• PGs below min_size, writes blocked. Recovery is the priority. There is no service to protect; restore redundancy fast.
• PGs undersized but above min_size, clients working. Client latency is the priority, within reason. The cluster is serving; the margin is reduced but real. Recover steadily rather than urgently.
• Backfill after adding capacity. No durability risk at all. This should be the slowest, most patient operation in the cluster, and there is no reason for it to be visible to clients.
• Deep scrub. Never urgent. Schedule it out of business hours and leave it alone.

That ordering is worth writing into the runbook, because it is the opposite of what pressure suggests. The situation that feels least urgent — adding a node — is the one people most often let run at full speed into a working day.

Measuring the trade

Watch both numbers at once, or the decision is blind:

ceph -s                # recovery rate: objects/s and MiB/s
ceph osd perf          # the latency cost being paid for it

If recovery is proceeding at a reasonable rate and client latency is inside budget, the balance is right. If recovery has stalled, the throttles are too low or something else is wrong. If client latency has doubled, the throttles are too high — regardless of how quickly recovery is finishing.

Quiz

Production discipline

Treat the recovery throttles as an incident-time control rather than a build-time setting, and change them while watching recovery rate and client p99 together. Rank the throughput-shaped activities by the durability risk they address: below min_size recovery outranks everything, undersized recovery is steady rather than urgent, and backfill after adding capacity has no claim on client latency at all. Put deep scrub in a window and leave it there.

Cross-course references

• Ceph: Part LX (Recovery Tuning) develops the individual settings.
• Ceph: Part LXI (Scrubbing) for the scrub schedule and its cost.
• Observability: alerting on client p99 during recovery is what makes this trade visible.