How long the rebuild actually takes

Why this matters in production

The exposure window is the rebuild duration, and somebody will ask for that number within the hour. An estimate built from a measured rate is defensible; one built from disk size divided by link speed is not.

Measuring what is left

ceph -s
ceph progress

  data:
    pgs:     3072 active+clean
             1024 active+undersized+degraded+remapped+backfilling
    objects: 12.57M objects, 47 TiB
  io:
    recovery: 1.9 GiB/s, 487 objects/s
  progress:
    Global Recovery Event (18m)
      [=====.......................] (remaining: 1h 47m)

ceph progress json | python3 -c '
import sys,json
for e in json.load(sys.stdin).get("events", []):
    print("%-40s %5.1f%%" % (e.get("message","?")[:40], e.get("progress",0)*100))'

ceph progress gives a remaining time. Treat it as a starting point: it extrapolates from recent rate and revises sharply when the rate changes.

The arithmetic

ceph -s --format json | python3 -c '
import sys,json
p = json.load(sys.stdin)["pgmap"]
deg = p.get("degraded_objects", 0)
tot = p.get("degraded_total", 1)
rate = p.get("recovering_objects_per_sec", 0)
print("degraded: %d of %d" % (deg, tot))
print("rate:     %d obj/s" % rate)
print("eta:      %.1f hours" % (deg / rate / 3600) if rate else "eta: stalled")'

Measure for ten minutes, extrapolate, then measure again. The rate is not
constant: the bulk moves fast and the last few PGs converge slowly,
because by then only a handful of OSD pairs are still working.

What actually bounds it

ceph osd df tree | sort -k17 -n | tail -5
ceph tell osd.3 bench 1073741824 4194304 2>/dev/null | head -4

Moving the balance

ceph config get osd osd_mclock_profile
ceph config set osd osd_mclock_profile high_recovery_ops

# the old knobs are ignored unless you say otherwise
ceph config set osd osd_mclock_override_recovery_settings true
ceph config set osd osd_max_backfills 4
ceph config set osd osd_recovery_max_active 8

# revert to the profile default when the rebuild is done
ceph config rm osd osd_max_backfills
ceph config set osd osd_mclock_override_recovery_settings false
ceph config set osd osd_mclock_profile balanced

Quiz

Production discipline

Build the estimate from degraded_objects and the observed recovery rate, re-measure after ten minutes, and quote a range rather than a time. If you change the mClock profile to shorten the window, write the revert into the same change record — a cluster left on high_recovery_ops pays client latency for every subsequent rebuild nobody is watching.

Cross-course references

• Kubernetes: a rollout ETA from replica counts ignores the slow tail of the last pods
• Linux: RAID rebuild speed is bounded by the surviving members, not the new disk