Backfill and recovery: two mechanisms, two triggers

Intermediate⏱ ~16 minceph

What you'll learn

Distinguish backfill from recovery precisely
Identify which is running from the PG states
Predict which a given event will trigger
Apply the right expectations to each

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Both produce data movement and both appear as recovery traffic in the status output. Confusing them means applying the wrong urgency — recovery is a durability problem and backfill usually is not.

The distinction

	Recovery	Backfill
Restores	missing copies	copies to their correct location
Triggered by	an OSD absent then returning, or replaced	topology change
Determines what to copy	from the PG log	by scanning the PG
Object state	degraded	misplaced
Urgency	durability at stake	optimisation
PG state	`recovering`	`backfilling`

Identifying which is running

ceph -s | grep -A6 'pgs:'

pgs: 3891 active+clean
     102  active+recovering+degraded      ← recovery
     360  active+remapped+backfilling      ← backfill

ceph -s | grep -E 'degraded|misplaced'
# 41213/18442104 objects degraded (0.223%)      ← recovery work
# 1841203/18442104 objects misplaced (9.98%)     ← backfill work

The two counts and the two PG states are independent signals of the same distinction.

Which event triggers which

Event	Triggers
OSD restarted briefly	recovery
OSD absent longer than the PG log window	backfill
OSD marked out	backfill of its data elsewhere
New OSD added	backfill into it
CRUSH rule changed	backfill
Pool `pg_num` increased	backfill
Device class reassigned	backfill
OSD replaced	backfill into the new one

Only the first row is recovery. Everything that changes placement produces backfill.

The expectations that follow

Recovery — the cluster is below its redundancy target. Expect it to be prioritised, expect it to be relatively quick if the log covered the gap, and treat a stall as urgent.

Backfill — the cluster has full copies in suboptimal locations. Expect it to take longer, expect it to yield to recovery, and treat a pause as acceptable.

ceph osd set norebalance     # pauses backfill, leaves recovery running

That flag exists precisely because pausing backfill is reasonable and pausing recovery is not.

Quiz

Knowledge check · 4 questions

Q1. A new OSD is added to a cluster. Does this trigger recovery or backfill?

Recovery, because copies are being created where none existed on that OSD Both simultaneously, since the PGs are degraded and misplaced at the same time Backfill, because the new OSD has no PG log position to compute a delta from Neither, until an OSD fails and the new one is needed to hold a replica
Q2. Pausing backfill with norebalance also pauses recovery.

True False
Q3. Set expectations for a pending expansion.

A cluster expansion will add 30 OSDs. Management asks whether the cluster will be at risk during the process and how long it will take.
Q4. Why is backfill slower per byte than recovery?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read the degraded and misplaced counts separately; they distinguish a durability problem from an optimisation one and warrant different urgency. Use norebalance when backfill needs pausing — it is the flag that defers the optimisation work without touching durability.

Cross-course references

Kubernetes: rebalancing pods across nodes versus recreating failed ones is the same distinction
Linux: a RAID reshape versus a rebuild differ in urgency for identical reasons