Deciding how to repair

Why this matters in production

There is more than one way to resolve an inconsistency and they have different consequences. Choosing by default means choosing ceph pg repair regardless of whether it is the right one.

The options

The criteria

flowchart TD
  A[Inconsistent PG] --> B{Clear majority?}
  B -->|yes| C{Device failing?}
  B -->|no| D{External evidence identifies good copy?}
  C -->|no| E[ceph pg repair]
  C -->|yes| F[Drain or replace OSD first, then repair]
  D -->|yes| G[Repair deliberately from that copy]
  D -->|no| H[Restore from backup]

The question that most often gets skipped is “is the device failing” — repairing onto a device that is about to produce the same error again means doing the work twice and leaving the PG exposed in between.

When repair is unsafe

# how many PGs are inconsistent, and where?
ceph health detail | grep 'pg .* inconsistent'
ceph pg dump pgs | grep inconsistent | awk '{print $1, $16}'

Several inconsistent PGs sharing an OSD points at that device. Several sharing a host but not an OSD points at memory, the HBA, or the cable path — and repairing from that host propagates the problem.

Sequencing with device replacement

# the device is failing: take it out of the acting set first
ceph osd out 12
# wait for the PG to be remapped and backfilled
watch -n 10 'ceph pg 3.1f query | grep -A2 acting'

# now the acting set holds only good copies
ceph pg repair 3.1f

# then replace the hardware
ceph orch osd rm 12 --replace

Draining first means the repair sources from a set that no longer contains the failing device, which is both safer and avoids repeating the work.

Quiz

Production discipline

Drain a failing device out of the acting set before repairing — repairing onto it repeats the work and leaves the PG exposed between. When inconsistencies span several OSDs on one host, drain the host before touching anything; the shared element is the host, not the disks.

Cross-course references

• Kubernetes: cordoning a suspect node before rescheduling follows the same ordering
• Linux: replacing the failing member before resyncing an array is standard practice