How recovery affects client I/O

Why this matters in production

Recovery competes with clients for exactly the same resources, so the question is never whether to accept impact but how much. Making that an explicit decision rather than a default is what keeps a recovery from becoming an incident.

The mechanism

Recovery and client I/O share:

The queue contention is the one that produces latency: a client operation arriving behind several recovery operations waits for them.

Measuring the impact

# baseline, before or between recoveries
ceph osd pool stats rbd-vms
rbd perf image iostat --pool rbd-vms

# during recovery
ceph -s | grep -E 'client|recovery'
ceph osd perf | sort -k2 -n | tail

# the ratio worth watching
ceph -s | grep -E 'client:|recovery:'
# client:   142 MiB/s rd, 88 MiB/s wr, 6.05k op/s
# recovery: 412 MiB/s, 103 objects/s

Recovery consuming several times the client bandwidth means clients are getting a fraction of the cluster.

Setting an acceptable level

The decision should be stated in the units clients experience:

acceptable: p99 write latency below 20 ms during recovery
target:     recovery completes within 12 hours

Both constraints together determine the throttle setting, and they may conflict — in which case one must give and the choice should be explicit.

ceph config set osd osd_mclock_profile high_client_ops   # favour clients
ceph config set osd osd_mclock_profile high_recovery_ops # favour recovery

The reasoning behind the default

Ceph defaults to favouring client I/O because redundancy above min_size means the cluster is safe while it recovers. That is correct while two copies remain and wrong once a PG is at one, which is why the decision should account for the actual exposure:

Quiz

Production discipline

Record the acceptable client latency and recovery window in the runbook so the throttle decision is a lookup rather than a judgment made under pressure. Invert the default priority when PGs are at a single copy — the reasoning behind favouring clients does not apply at that exposure.

Cross-course references

• Kubernetes: priority classes ordering competing work is the same trade made explicit
• Linux: ionice for background maintenance addresses the identical contention