Estimating how long a backfill will take

Intermediate⏱ ~16 minceph

What you'll learn

Estimate backfill duration from observable figures
Account for the factors that change the rate
Refine an estimate as it progresses
Communicate an estimate honestly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“How long?” is the first question asked and the one most often answered badly. A defensible estimate comes from measurement rather than from the data volume alone.

The arithmetic

ceph -s | grep -E 'misplaced|recovery'
# 1841203/18442104 objects misplaced (9.98%)
# recovery: 412 MiB/s, 103 objects/s

1841203 objects / 103 objects per second ≈ 17880 seconds ≈ 5 hours

misplaced=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
rate=$(ceph pg stat | grep -oE '[0-9]+ objects/s' | head -1 | grep -oE '^[0-9]+')
echo "$((misplaced / rate / 3600)) hours at the current rate"

The progress module

ceph progress
# Global Recovery Event (2h)
#   [==============..............] (remaining: 3h)

This is the estimate to quote, because it accounts for the rate it has actually observed rather than an instantaneous sample.

Sources of error

Factor	Effect on the estimate
Rate varies with client load	overnight is faster than midday
Throttle changes	any adjustment invalidates the estimate
A failure during the backfill	recovery takes priority, pausing it
`backfill_toofull`	stalls entirely
Object size variation	objects per second is not a constant volume

The last is worth noting: a pool with mixed object sizes moves at a variable byte rate even at a constant object rate, so a bytes-based estimate and an objects-based one can disagree.

Refining as it progresses

# sample the rate over a window rather than instantaneously
a=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
sleep 900
b=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
echo "$(( (a-b) / 900 )) objects/s over 15 minutes"
echo "$(( b / ((a-b)/900) / 3600 )) hours remaining"

A fifteen-minute sample is far more reliable than an instantaneous rate, which fluctuates.

Communicating honestly

Poor	Better
“About five hours”	“Five hours at the current rate; overnight it will be faster and it pauses if an OSD fails”
“It should be done tomorrow”	“Currently projecting completion tomorrow afternoon; I will update at 09:00”
“Hard to say”	“Between eight and twenty hours depending on load; the estimate narrows as it progresses”

A range with the factors that move it is more useful than a point estimate that will be wrong.

Quiz

Knowledge check · 4 questions

Q1. Why is an estimate taken in the first minutes of a backfill usually too pessimistic?

The misplaced count is inaccurate until every PG has finished peering and reported statistics The progress module is not initialised until the event has been running for a full hour Client load peaks at the start because the topology change flushes every client's cached map Concurrency ramps up as queued PGs start and more OSDs participate, so the rate rises
Q2. A fifteen-minute measurement of the misplaced count gives a more reliable rate than the instantaneous figure in `ceph -s`.

True False
Q3. Give a defensible estimate for a large backfill.

Management asks when a cluster expansion backfill will complete. It started twenty minutes ago and `ceph -s` currently shows a rate that projects eleven hours.
Q4. Why can a bytes-based estimate and an objects-based one disagree?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure the rate over a fifteen-minute window before quoting an estimate; the instantaneous figure fluctuates enough to be out by a factor of two. Quote a range with the factors that move it and commit to an update time — a point estimate that turns out wrong costs more credibility than a range that holds.

Cross-course references

Kubernetes: rollout completion estimates face the same ramp and variance
Linux: any long-running background job estimate benefits from windowed measurement