Replacing a failing disk before it fails

Why this matters in production

The same replacement done proactively costs one planned drain; done reactively it costs an unplanned recovery, a degraded window, and often a scrub inconsistency.

Recognising the device

DEVICE_ID=12
ceph device ls
ceph health detail | grep -i DEVICE_HEALTH
ceph device predict-life-expectancy ${DEVICE_ID}

DEVICE_ID=12
ceph device get-health-metrics ${DEVICE_ID} | python3 -c '
import sys,json; d=json.load(sys.stdin)
for ts in sorted(d)[-3:]:
    a = d[ts].get("ata_smart_attributes",{}).get("table",[])
    p = {x["id"]: x["raw"]["value"] for x in a if x["id"] in (5,197,198)}
    print(ts, p)'

Three samples showing a rising count is the confirmation.

Planning the window

A proactive replacement needs:
  the cluster healthy, with no other work in progress
  capacity for the drain
  a replacement drive on hand
  a window when the backfill impact is acceptable

ceph -s
ceph osd df | awk -v o=44 '$1==o {print $7, "used"}'
ceph osd df | sort -k17 -rn | head -3

Unlike a failure, this can wait for the right moment — which is the whole benefit.

Executing

# 1. drain cleanly; the device still reads
ceph orch osd rm 44 --replace
ceph orch osd rm status

# 2. wait for it to complete
watch -n 30 'ceph orch osd rm status; ceph -s | grep -E "misplaced|health"'

# 3. confirm zero PGs and cluster health
ceph pg ls-by-osd 44 2>/dev/null | tail -n +2 | wc -l
ceph -s | grep HEALTH_OK

# 4. replace the drive and let the orchestrator redeploy
ceph orch device ls ceph-03 --refresh

The drain completing before the drive is removed means no PG is ever degraded.

Comparing the costs

The device cost is the same either way.
Everything else is worse in the reactive case.

Quiz

Production discipline

Replace on the health trigger rather than on failure — the drive costs the same and the proactive drain means no PG is ever degraded. Confirm the rising trend across several health samples, then choose a window when the backfill impact is acceptable.

Cross-course references

• Kubernetes: draining a node before it fails avoids the disruption of it failing
• Linux: proactive RAID member replacement has the identical cost comparison