Deciding between repair and restore

Advanced⏱ ~18 mincephrados

What you'll learn

Apply the four questions that decide repair or restore
Bound the corruption window against backup ages
Preserve evidence before an irreversible action
Choose regeneration where it is available

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Repair overwrites the copies it judges wrong and cannot be undone. Restoring to a new target changes nothing that already exists. The two options are not symmetric, and the cheaper one is the destructive one.

The four questions

1. Is there a copy that can be shown to be correct?
2. Is the corruption confined, or spread across devices and hosts?
3. When did it start, relative to the newest backup?
4. Can a producer regenerate the data more cheaply than either option?

Bounding the corruption window

rados -p rbd stat rbd_data.2ae94b8b4567.0000000000000a3c

rbd/rbd_data.2ae94b8b4567.0000000000000a3c mtime 2026-08-11 04:19:52.000000, size 4194304

ceph pg 12.1a4 query | python3 -c '
import sys,json
s = json.load(sys.stdin)["info"]["stats"]
print("clean at:", s["last_deep_scrub_stamp"])'
ceph config get osd osd_deep_scrub_interval

Known	Bound it gives
last clean deep scrub	the corruption is newer than this
object mtime	the content is no older than this
deep-scrub interval	how wide the window is when scrubs are on schedule
overdue deep-scrub warnings	how much wider it actually is

Any backup taken inside that window may contain the corrupt object. The
first backup older than the window is the first one you can trust — which
makes retention depth a data integrity parameter, not only a cost one.

The decision table

Situation	Action
one shard `read_error`, others agree	repair
size 2, digests differ, clean reads	corroborate externally first
many PGs, one device	fail the device out, then repair the affected objects
many PGs, many devices on one host	diagnose the host before touching anything
corruption predates the newest backup	validate an older backup before restoring
EC pool, bad shards exceed `m`	restore
data is derived from a producer	regenerate

Preserving evidence first

ceph orch daemon stop osd.8
cephadm shell --name osd.8 -- ceph-objectstore-tool \
  --data-path /var/lib/ceph/osd/ceph-8 --pgid 12.1a4 \
  "$OBJ" get-bytes > /secure/incident/shard-8.bin
ceph orch daemon start osd.8

rados list-inconsistent-obj 12.1a4 --format=json-pretty \
  > /secure/incident/12.1a4-shards.json
ceph pg 12.1a4 query > /secure/incident/12.1a4-query.json

Regeneration, where it applies

ceph df
ceph osd pool ls detail | grep -E '^pool'

Derived pools — build caches, rendered tiles, analytics intermediates, replica indexes — are cheaper to rebuild than to adjudicate. Deciding that before the incident is what makes the decision fast during one.

Why the corruption window decides which backup is usable

Corruption is discovered at scrub time, not at write time. With the default deep-scrub interval, an error surfacing today could have been written any time in the past week — longer if the cluster has been warning about PGs not deep-scrubbed in time. Every backup taken inside that window copied whatever was on disk, corruption included, and a restore from one of them reinstates the damage while appearing to succeed. The first trustworthy backup is the first one predating the whole window, so the usable recovery point is not the newest backup but the newest one older than the corruption. A retention policy of three daily copies cannot survive a week-wide window, which is why deep-scrub coverage and backup retention have to be designed against each other.

Quiz

Knowledge check · 4 questions

Q1. Why is the choice between repair and restore asymmetric?

Repair is slower than a restore, because it must re-read every object in the affected placement group before it can rewrite any of them Restore requires taking the pool offline for the duration, while a repair runs against a live pool without interrupting clients Repair only works on replicated pools, so an erasure-coded pool has no reversible option available to it and must always be restored Repair overwrites the copies it rejects and cannot be undone, while a restore to a new target adds data and leaves the damaged original available for analysis
Q2. An inconsistency reported today means the corruption happened today.

True False
Q3. Decide how to recover a corrupt object.

A single object in a size 3 pool has one shard with a digest mismatch and no read error; the other two agree. Backups run nightly with three days of retention, and the PG was last deep-scrubbed eleven days ago.
Q4. What makes backup retention depth a data integrity parameter?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Extract every shard before repairing anything you cannot adjudicate — repair destroys the evidence that would let you discover you chose wrongly. Compare the corruption window against your retention depth before treating any backup as clean; a nightly backup with three days of retention cannot span a week-wide window.

Cross-course references

Kubernetes: restoring an etcd snapshot taken after the bad change restores the bad change
Linux: an irreversible fix and a reversible one are not interchangeable at equal confidence