The inconsistent PG alert and its handling

Why this matters in production

PG_DAMAGED sounds like an emergency and usually is not. Treating it as one burns credibility; treating it as routine leaves corrupt data readable.

Setting the urgency

Not an outage:   the PG is active+clean+inconsistent and serving I/O
Not routine:     a client can read the bad copy at any time
Correct urgency: investigate within the hour, repair within the day

ceph health detail | grep -A3 PG_DAMAGED
ceph -s | grep -c 'scrub errors'

The investigation sequence

# 1. what is inconsistent?
PGID=12.1a
OSD=12
DEVICE_ID=12
rados list-inconsistent-obj ${PGID} --format=json-pretty

# 2. which shard, and what error?
rados list-inconsistent-obj ${PGID} --format=json-pretty | python3 -c '
import sys,json; d=json.load(sys.stdin)
for i in d["inconsistents"]:
    print(i["object"]["name"], i.get("union_shard_errors"))
    for s in i["shards"]:
        print("   osd.%s %s digest=%s" % (s["osd"], s.get("errors"), s.get("data_digest")))'

# 3. the device evidence
ceph osd find ${OSD}
ceph device get-health-metrics ${DEVICE_ID}

Three steps produce the diagnosis: which object, which shard disagrees, and whether that shard’s device explains it.

The repair decision

PGID=12.1a
ceph pg repair ${PGID}
watch -n 15 'ceph pg ${PGID} query | grep -E "\"state\""'

Closing the loop

A repair that is not followed by device action leaves the cause in place:

# after the repair
DEVICE_ID=12
PGID=12.1a
ceph device get-health-metrics ${DEVICE_ID} | tail -20
ceph pg deep-scrub ${PGID}

Closure checklist:
  - the PG returns to active+clean
  - a confirming deep scrub passes
  - the affected object is validated at the application level if it was a digest mismatch
  - the device is replaced if SMART supports it
  - the event is recorded with the device serial

The last item is what makes a second event on the same device recognisable.

Quiz

Production discipline

Treat PG_DAMAGED as investigate-within-the-hour, repair-within-the-day — it is neither an outage nor something to defer. Close the loop with device action and record the device serial; a repair without it guarantees the same error at the next deep scrub.

Cross-course references

• Kubernetes: fixing a symptom without addressing the node that caused it recurs identically
• Linux: repairing filesystem corruption without replacing the failing disk repeats