When repair does not repair

Advanced⏱ ~18 mincephrados

What you'll learn

Confirm a requested repair actually started
Diagnose repairs that recur after completing
Recognise that repair correctly abstains on some objects
Verify the outcome rather than the command

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

ceph pg repair returns instantly and says nothing about what happened. Everything that matters — whether it started, whether it fixed anything, whether it will happen again — is established afterwards.

Confirming it started

ceph pg repair 12.1a4
ceph pg ls inconsistent

PG      OBJECTS  STATE                                          ACTING
12.1a4      2841  active+clean+scrubbing+deep+inconsistent+repair  [17,42,8]

Reason a repair does not start	Check
cluster scrub flags set	the flags line of `ceph osd dump`
the OSD is at its scrub concurrency limit	`ceph config get osd osd_max_scrubs`
the PG is not active+clean	state includes `degraded` or `backfilling`
the primary changed after the request	re-issue against the current primary

ceph osd dump | grep -E '^flags'
ceph config get osd osd_max_scrubs
ceph status

A request against a PG whose state never gains `repair` was queued and
starved, not executed. Watch for the state, not for the command exiting.

Repairs that recur

Symptom	What is happening
new errors, different objects, same OSD	the device is still failing
the same object again after repair	the writer is still producing bad data
errors spreading to new PGs	the fault is above the device
errors return only after recovery	a bad copy is being propagated by backfill

ceph pg ls-by-osd 17 | head
ceph device ls-by-daemon osd.17
ceph log last 100 info cluster | grep -iE 'repair|inconsistent'

When repair correctly abstains

Where no copy can be selected, repair completes and leaves the object inconsistent. That is the right outcome — the alternative would be overwriting one unverified copy with another.

rados list-inconsistent-obj 12.1a4 --format=json-pretty | python3 -c '
import sys,json
d = json.load(sys.stdin)
for o in d.get("inconsistents", []):
    print(o["object"]["name"], o.get("errors"), o.get("union_shard_errors"))'

Verifying the outcome

ceph pg deep-scrub 12.1a4

# wait for the stamp to advance before believing anything
ceph pg 12.1a4 query | python3 -c '
import sys,json
print(json.load(sys.stdin)["info"]["stats"]["last_deep_scrub_stamp"])'

rados list-inconsistent-obj 12.1a4 --format=json-pretty | python3 -c '
import sys,json
print(len(json.load(sys.stdin).get("inconsistents", [])), "remaining")'
ceph health detail | grep -E 'OSD_SCRUB_ERRORS|PG_DAMAGED'

Why automatic repair stops above a threshold

osd_scrub_auto_repair lets a scrub fix what it finds without an operator request, but only while the number of errors in that scrub stays at or below osd_scrub_auto_repair_num_errors. The threshold exists because the two situations look identical to a single PG and are entirely different at cluster scale: a handful of errors is what isolated media defects produce, while dozens are what a dying device, a flaking HBA or bad memory produce — and in that second case the copies repair would use as sources came off the same suspect hardware. Ceph declines to guess, raises the health error and waits for a human to establish the correlation first. Overriding the threshold to make a large error count disappear automates exactly the mistake it was written to prevent.

Quiz

Knowledge check · 4 questions

Q1. What is the only reliable evidence that a repair fixed a PG?

The `ceph pg repair` command returning without error, since the primary accepted the request A subsequent deep scrub whose timestamp has advanced and which reports no errors An empty `rados list-inconsistent-obj`, which queries the OSDs for their current state The PG returning to active+clean, which it cannot do while inconsistent objects remain
Q2. A repair that completes and leaves the object inconsistent means the command failed.

True False
Q3. Handle a repair that keeps recurring.

A PG is repaired successfully on Monday. On Wednesday it is inconsistent again, with different object names, and two other PGs sharing OSD 17 have also become inconsistent.
Q4. Why does automatic repair stop above an error threshold?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Watch for the PG state to gain repair rather than trusting the command’s exit — a starved request looks identical to a successful one. Verify with a fresh deep scrub whose timestamp has advanced; the inconsistent-object list is the previous scrub’s cached record and will happily agree with you.

Cross-course references

Kubernetes: a controller accepting a spec change is not evidence the change reconciled
Linux: cached status output confirms a past state, not the current one