The client versus recovery trade-off

Why this matters in production

There is no setting that makes recovery fast and free. The devices and the network are finite, and recovery consumes what clients would otherwise use. Tuning is choosing a point on that curve, not escaping it.

What is traded

The degraded window is the period during which a further failure could cause data loss or unavailability. That is the quantity being traded for latency, and stating it that way changes the conversation.

The curve is not linear

ceph config set osd osd_max_backfills 1     # baseline
ceph config set osd osd_max_backfills 2     # roughly 2x recovery, small latency change
ceph config set osd osd_max_backfills 8     # marginal recovery gain, large latency change

Devices have headroom, and consuming it costs little until it runs out — after which latency rises steeply. The useful position is at the knee of that curve, which is found by measurement rather than by choosing a number.

Choosing a position deliberately

Three positions and when each is right:

# recovery first
ceph config set osd osd_mclock_profile high_recovery_ops

# clients first
ceph config set osd osd_mclock_profile high_client_ops

# balanced
ceph config set osd osd_mclock_profile balanced

The position should follow from the redundancy state, not from a standing preference.

Communicating the choice

The trade is legible to non-operators when framed as exposure:

Recovery will complete in four hours with client latency at roughly twice baseline, or in sixteen hours at close to baseline. During the recovery a second disk failure in the same rack would take the affected pool offline. Which window do you want?

That is a question stakeholders can answer. “Should I raise osd_max_backfills?” is not.

Quiz

Production discipline

Set the tuning position from the redundancy state — recovery-first when a further failure would breach min_size, clients-first during a planned expansion — and reset it when the state changes. Present the trade to stakeholders as exposure duration against latency, which is a question they can answer.

Cross-course references

• Kubernetes: pod disruption budgets encode the same exposure-versus-velocity trade
• Linux: RAID rebuild rate limits present exactly this choice