Validating a replacement after deployment

Why this matters in production

A replacement that is up and receiving data may still be wrong in ways that surface weeks later.

The immediate checks

ceph osd tree | grep -A3 ceph-03
ceph osd df | awk '$1==44'
ceph osd metadata 44 | python3 -c '
import sys,json; d=json.load(sys.stdin)
for k in ("hostname","device_ids","bluestore_bdev_type","osd_objectstore"):
    print("%-28s %s" % (k, d.get(k)))'

ceph osd crush get-device-class osd.44
ceph osd df | awk '$1==44 {print "weight", $4, "size", $5}'

Comparing performance

# per-OSD benchmark, against its peers
for osd in $(ceph osd ls-tree ceph-03); do
  printf 'osd.%-4s ' "$osd"
  ceph tell osd.$osd bench 12288000 4096 4194304 100 2>/dev/null | \
    python3 -c 'import sys,json; print(round(json.load(sys.stdin)["iops"],1), "IOPS")'
done

# and the latency it reports in service
ceph osd perf | awk '$1==44 || NR==1'
ceph osd perf | awk 'NR>1 {s+=$2; n++} END {print "cluster avg", s/n}'

A replacement performing materially below its peers is a finding, whether the cause is the drive, its firmware, or its position.

Confirming the cluster state

ceph -s
ceph health detail

# the backfill completed
ceph -s | grep -E 'misplaced|degraded' || echo "clean"

# the utilisation spread
ceph osd df | awk 'NR>1 {if($17+0>m)m=$17+0; if(mn==""||$17+0<mn)mn=$17+0}
  END {printf "spread %.1f\n", m-mn}'

# and no residue from the removal
ceph osd tree | grep -E 'destroyed|DNE'

# a confirming scrub on PGs that were on the old device
PGID=12.1a
ceph pg ls-by-osd 44 | head -5
ceph pg deep-scrub ${PGID}

Recording it

2026-08-18  osd.44  ceph-03  slot 6
  Removed:  HGST HUH721616ALE600 serial 2EJXYZ1A
            reason: 74 pending sectors, rising
            age: 3.2 years
  Fitted:   HGST HUH721616ALE600 serial 2FKLMN2B firmware LEGNW9G0
  Verified: class hdd, weight 14.55190, up+in, backfill complete 2026-08-18 19:40
            bench 51 IOPS against a fleet median of 49

The removed drive’s serial, age, and reason are what build the fleet failure model over time.

Quiz

Production discipline

Benchmark a replacement against its peers with ceph tell osd bench before considering it validated — an outlier found now is a warranty conversation rather than a latency investigation later. Record the removed drive’s serial, age, and failure reason; that record is the fleet failure model.

Cross-course references

• Kubernetes: validating a replaced node against its peers before returning it to service
• Linux: post-replacement verification is what distinguishes a fix from an assumption