Deciding when to replace a device

Intermediate⏱ ~17 mincephsmartctl

What you'll learn

Define replacement triggers
Justify proactive replacement economically
Sequence a replacement safely
Track fleet age and failure patterns

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A device replaced on a warning costs one planned drain. The same device replaced after failure costs an unplanned recovery, possible inconsistencies, and whatever the degraded window exposed.

The triggers

Trigger	Action
Current pending sectors > 0	replace
Reallocated sectors increasing	replace
Offline uncorrectable > 0	replace
NVMe `critical_warning` non-zero	replace
NVMe `percentage_used` > 90	plan replacement
`available_spare` below threshold	replace
Repeated scrub errors on the same OSD	replace
Ceph `DEVICE_HEALTH` prediction	plan replacement
Age beyond the fleet policy	plan replacement
Latency persistently above class	investigate, then replace

The last row needs the investigation first: a slow OSD can be a host problem, and replacing a healthy drive fixes nothing.

The economic case

proactive replacement:
  one planned drain (hours of backfill, no degradation)
  one device cost
  scheduled, during working hours

reactive replacement:
  unplanned recovery (degraded window, elevated risk)
  possible scrub inconsistencies to repair
  possible client impact
  same device cost
  unscheduled, often out of hours

The device cost is identical; everything else is worse. That is the whole argument, and it holds regardless of how much life the drive might have had left.

Sequencing a replacement

# 1. verify the cluster can absorb it
ceph osd df | sort -k17 -rn | head -3
ceph -s

# 2. drain
ceph osd out 44
watch -n 30 'ceph pg ls-by-osd 44 | wc -l'

# 3. remove, marking the ID for reuse
ceph orch osd rm 44 --replace

# 4. physically replace, then let cephadm redeploy
ceph orch device ls --refresh
ceph -s

--replace preserves the OSD ID so the new device takes the same identity, which keeps the CRUSH placement stable and avoids a second rebalance.

Tracking fleet patterns

ceph device ls --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    print(d.get("devid"), d.get("location",[{}])[0].get("host"), d.get("daemons"))'

Fleet record:
  Model            Count  Deployed  Failures  MTBF observed
  HGST HUH721616   96     2024-03   4         ~5.2 years
  Samsung PM9A3    24     2025-01   0         —
  Seagate ST16000  48     2023-08   11        ~2.1 years    ← investigate

A model failing well below its cohort is a purchasing decision, and the data to see it only exists if replacements are recorded.

Quiz

Knowledge check · 4 questions

Q1. Why does `ceph orch osd rm --replace` halve the data movement of a device replacement?

It copies the old device's contents straight onto the replacement, so only writes made during the swap need backfilling It skips the drain entirely and lets the replacement backfill from the surviving replicas once it is inserted It defers the resulting rebalance to an off-peak window, so half the movement happens outside the replacement itself Preserving the OSD ID means CRUSH sees one map transition instead of two, so the PGs return to the same OSD
Q2. Proactive replacement is harder to justify economically because the drive might have had life remaining.

True False
Q3. Establish a replacement policy.

A cluster replaces drives only after they fail. Over the past year this produced six unplanned recoveries, four of which coincided with scrub inconsistencies requiring repair.
Q4. Why should a latency outlier be investigated before the device is replaced?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Replace devices on the trigger rather than on failure — the device cost is the same and reactive replacement adds a degraded window and likely inconsistencies. Use ceph orch osd rm --replace to preserve the OSD ID, which halves the data movement, and record every replacement so fleet failure patterns become visible.

Cross-course references

Kubernetes: draining a node before it fails follows the identical economics
Linux: proactive RAID member replacement is long-standing practice for the same reasons