What recovery does and how it knows what to copy

Why this matters in production

Recovery is targeted: it copies exactly the objects that are missing and nothing else. That precision is what makes a brief OSD absence cheap, and understanding its limit explains why a long absence is not.

The mechanism

1. A PG has fewer current copies than size
2. Peering establishes which objects each peer holds and at what version
3. The primary computes the missing set from the PG logs
4. It copies those objects to the peers that lack them
5. The PG returns to active+clean

ceph pg 7.3d query | jq -r '.info.stats.stat_sum.num_objects_recovered'
ceph -s | grep recovery

How the missing set is computed

Every write is appended to the PG log with a version:

41207'8823910  MODIFY  rbd_data.1f2a3b.0000000000000c17
41207'8823911  MODIFY  rbd_data.1f2a3b.0000000000000c18
41207'8823912  DELETE  rbd_data.1f2a3b.0000000000000a02

A returning peer reports its last_update. The primary compares it against its own and the difference is exactly the set of objects to copy.

ceph pg 7.3d query | jq -r '.peer_info[] | {peer, last_update: .last_update}'

The limit

The log is bounded:

ceph config get osd osd_min_pg_log_entries      # 100
ceph config get osd osd_max_pg_log_entries      # 10000

If the peer’s position has been trimmed off the log, the delta cannot be computed and the PG falls back to backfill — a full scan and copy.

Predicting the volume

A 30-second OSD restart: the writes that occurred in 30 seconds. Trivial.

A one-hour outage on a busy pool: an hour of writes, if the log spans it. Otherwise backfill of the whole OSD.

A new OSD: always backfill — there is no log position to compare.

# how much changed while an OSD was absent
ceph pg dump | awk '{print $1, $2}' | head
ceph -s | grep -E 'degraded|misplaced'

Quiz

Production discipline

Consider raising osd_max_pg_log_entries on clusters with frequent short OSD restarts — rolling upgrades, flapping hardware — since it keeps those absences on the cheap recovery path. Weigh it against OSD memory, which the log consumes per PG.

Cross-course references

• Kubernetes: incremental resync versus full relist follows the same bounded-history logic
• Linux: a bounded journal determining incremental versus full recovery is a recurring design