Preventing inconsistencies

Why this matters in production

Most inconsistencies are produced by devices that gave warning first. The warning is available and is usually not being watched.

Ceph’s device monitoring

DEVICE_ID=12
ceph device ls
ceph device ls-by-host ceph-osd-03
ceph device get-health-metrics ${DEVICE_ID}
ceph device predict-life-expectancy ${DEVICE_ID}

# enable collection
ceph config set mgr mgr/devicehealth/enable_monitoring true
ceph config set mgr mgr/devicehealth/scrape_frequency 86400

With monitoring enabled, Ceph collects SMART data daily and raises a health warning when a device is predicted to fail:

ceph health detail | grep -i DEVICE_HEALTH

Replacement thresholds

Waiting for a device to fail means accepting the inconsistencies it produces on the way. Useful thresholds:

for h in $(ceph orch host ls --format json | python3 -c 'import sys,json; [print(x["hostname"]) for x in json.load(sys.stdin)]'); do
  echo "== $h"
  ceph device ls-by-host "$h"
done

The threshold that matters most is any pending sectors: a pending sector is a block the device could not read and has not yet been able to reallocate, which is precisely the condition that produces a read_error on the next scrub.

Configuration that limits exposure

ceph config set osd osd_scrub_auto_repair true
ceph config set osd osd_scrub_auto_repair_num_errors 5

osd_scrub_auto_repair repairs automatically when the error count is below the threshold, which covers the routine single-object read error while still surfacing anything larger for a human.

A device lifecycle practice

1. record the SMART baseline at deployment
2. collect daily via ceph device
3. alert on any pending or reallocated sector increase
4. drain and replace on the alert, not on failure
5. record the device's service life for the fleet model

The fourth step is the one with the return: a device drained on a warning produces no inconsistencies at all.

Quiz

Production discipline

Enable mgr/devicehealth monitoring and alert on any increase in pending or reallocated sectors, then drain on the alert rather than on failure — a device drained on a warning produces no inconsistencies at all. Prefer size=3 over size=2 so every repair decision has a majority.

Cross-course references

• Kubernetes: node problem detector draining before failure is the same practice
• Linux: SMART-triggered proactive replacement is standard fleet discipline