The cost of the partition healing

Why this matters in production

The partition ending is not the incident ending. What follows is a bulk data movement whose size was decided ten minutes into the outage, by a timer nobody was watching, and whose rate is governed by a scheduler most operators have never configured.

The movement you chose without noticing

mon_osd_down_out_interval defaults to 600 seconds. An OSD still down at that point is marked out, CRUSH recomputes without it, and the cluster begins rebuilding its data elsewhere. When the partition heals, the OSD comes back in and CRUSH recomputes again — moving much of that data back.

ceph osd set noout
ceph osd set nobackfill      # optional, if you want to stage the return

Setting noout inside the first ten minutes of any suspected partition is the single highest-leverage action available, and it is free if the hardware turns out to be genuinely dead — you unset it and the normal rebuild proceeds.

Staging the rejoin

ceph osd set noup
systemctl start ceph-$FSID@osd.42.service
ceph osd unset noup

A rack of OSDs rejoining simultaneously commits map epochs faster than peering completes, and every epoch restarts peering for the affected PGs. Where the returning set is large, bring it back in groups and let ceph -s settle between them.

Rate control under mClock

Tentacle uses the mClock scheduler by default, and mClock — not the classic recovery settings — governs the split between client and recovery work.

ceph config get osd osd_op_queue
ceph config get osd osd_mclock_profile
ceph config set osd osd_mclock_profile high_client_ops

# osd_max_backfills and friends are ignored unless this is set
ceph config get osd osd_mclock_override_recovery_settings

Setting osd_max_backfills on a Tentacle cluster and observing no change is the most common recovery-tuning surprise — the value is accepted and disregarded until osd_mclock_override_recovery_settings is true.

Watching it complete

ceph -s
ceph progress
ceph pg dump_stuck inactive
ceph pg dump_stuck degraded

  io:
    recovery: 1.8 GiB/s, 462 objects/s

  progress:
    Global Recovery Event (18m)
      [=============---------------] (remaining: 24m)

The remaining estimate is an extrapolation of the current rate and it will move as the rate does. Track the degraded object count directly if you owe someone a number.

Confirming what did not come back

ceph health detail
ceph pg dump_stuck stale
ceph osd tree down

An OSD that stays down after the network returns has a second, unrelated fault that the partition was hiding. Objects reported unfound need the specific PG queried before anything else is done to them.

Quiz

Production discipline

Set noout in the first ten minutes of any suspected partition; it is reversible, free when the loss is real, and it halves the data movement when it is not. When the network returns, stage the rejoin with noup rather than starting everything at once, and reach for osd_mclock_profile rather than osd_max_backfills, which Tentacle ignores by default.

Cross-course references

• Kubernetes: pod eviction timeouts create the same evacuate-then-return churn
• Linux: a scheduler with a default policy silently overrides the knob you set