Slow ops caused by recovery

Why this matters in production

Slow ops during a recovery may be the recovery’s cost or may be a fault the recovery exposed. Throttling addresses the first and hides the second.

Confirming the attribution

# the definitive test
ceph osd set norebalance
ceph osd set norecover
sleep 300
ceph health detail | grep -c SLOW_OPS
ceph osd unset norecover
ceph osd unset norebalance

If the slow ops clear during the pause and return afterwards, recovery is the cause. If they persist through the pause, something else is responsible and recovery merely coincided.

# and the correlation over time
ceph -s | grep -E 'recovery|slow'

Deciding whether to act

The redundancy state determines the answer. Throttling recovery while a pool sits at min_size prolongs the window in which a further failure causes an outage.

ceph pg dump pgs | awk '$10 ~ /undersized/ {print $1, $10}' | head
ceph osd pool get rbd-vms min_size

Choosing the throttle

# the primary lever under mClock
ceph config set osd osd_mclock_profile high_client_ops

# pause the optimisation work only
ceph osd set norebalance

# reduce concurrency, if not under mClock
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active_hdd 1

norebalance is the right choice for an expansion: it stops the misplaced-object movement while leaving degraded-PG recovery running.

Recovery exposing a fault

recovery adds sustained load
  → a marginal device is pushed past its comfortable operating point
  → its latency rises sharply
  → slow ops appear
  → the recovery is blamed

# the distinguishing evidence
ceph osd perf | sort -k2 -rn | head -5

If the slow ops concentrate on one or two OSDs while recovery is spread across the cluster, the concentration is the fault. Recovery load is distributed; a marginal device is not.

Quiz

Production discipline

Pause recovery for five minutes and re-measure before attributing slow ops to it; correlation alone cannot distinguish recovery causing the delay from recovery exposing a marginal device. Let the redundancy state decide whether to throttle — at min_size, accelerating is correct and throttling prolongs the exposure.

Cross-course references

• Kubernetes: background reconciliation exposing an under-provisioned node behaves the same
• Linux: a RAID rebuild revealing a second marginal member is the classic analogue