Writes while a replica is missing

Why this matters in production

The whole point of size 3, min_size 2 is that you keep serving through a failure. But “keep serving” raises an immediate question: if a replica is missing while writes continue, how does it ever catch up? The answer — the PG log — also explains why a brief OSD outage is cheap and a long one is expensive.

Writing with a replica down

An OSD fails. Its PGs go active+undersized+degraded. Two copies remain, which meets min_size, so:

• The primary accepts the write
• It replicates to the one surviving secondary
• Both commit
• The client is acknowledged

The write is durable on two OSDs. The pool’s size is 3, so the cluster knows it owes one more copy, and that debt is what degraded means.

The PG log

Every write is appended to the PG log — an ordered record of recent modifications, each with a version number of the form epoch'seq:

41207'8823910  MODIFY  rbd_data.1f2a3b.0000000000000c17
41207'8823911  MODIFY  rbd_data.1f2a3b.0000000000000c18
41207'8823912  DELETE  rbd_data.1f2a3b.0000000000000a02

When the missing OSD returns, peering compares its last_update against the primary’s. The difference is exactly the set of objects it needs. The primary sends those objects, and only those.

ceph pg 7.3d query | jq -r '.info.stats.last_update, .info.log_tail'

This is recovery: targeted, proportional to how much changed while the OSD was away.

When the log is not enough

The log is bounded — roughly osd_max_pg_log_entries entries per PG. If the OSD was away long enough that writes pushed its position off the tail of the log, the primary can no longer compute the delta. It falls back to backfill: scan the entire PG and copy everything.

This is the concrete reason a 30-second OSD restart is nearly free and a two-hour outage is a night of backfill: the difference is whether the log still spans the gap.

Quiz

Production discipline

Set noout before any planned host maintenance so the outage stays on the recovery side of the line, and clear it immediately afterwards. Alert on degraded object counts rather than on OSD-down counts — the former measures your actual exposure, the latter only its cause.

Cross-course references

• Kubernetes: a Pod rescheduled quickly reuses its PVC; one gone long enough forces a full re-sync in the application
• Linux: this is the same distinction as an incremental rsync versus a full copy