Marking out and the two data movements

Why this matters in production

Marking an OSD out and removing it from CRUSH both trigger backfill, and doing them at different times makes the cluster move data twice for one failure.

Two different weights

ceph osd tree | head -8
ceph osd dump | grep -E '^osd[.]18 '

ID  CLASS  WEIGHT   TYPE NAME     STATUS  REWEIGHT
18   ssd   7.27664      osd.18     down          0

ceph osd out 18 19 20            # sets REWEIGHT to 0
ceph osd in 18 19 20             # sets it back to 1

ceph osd crush reweight osd.18 0 # changes the CRUSH weight — much larger

Watching the movement

ceph -s | grep -E 'degraded|misplaced|recovering|backfill'

Degraded falling and misplaced rising is recovery working. Misplaced
rising with degraded flat means something changed placement, not
redundancy.

Reversing it

ceph osd in 18 19 20
ceph -s | grep misplaced

Marking back in after backfill has partly completed does not undo the
work already done. The copies made elsewhere stay until the returning
OSDs are current, then the surplus is removed.

# check the returning OSDs are actually healthy before marking in
ceph osd metadata 18 | python3 -c '
import sys,json
d = json.load(sys.stdin)
for k in ("hostname","osd_objectstore","bluestore_bdev_type","default_device_class"):
    print("%-24s %s" % (k, d.get(k)))'
ceph osd perf | head

Drain or wait

ceph orch host drain ceph-osd-07
ceph orch osd rm status

Quiz

Production discipline

Treat CRUSH removal as a scheduled change with its own backfill, not as cleanup after recovery. When a host is confirmed dead, drain it in one pass so the cluster pays for the failure once rather than twice.

Cross-course references

• Kubernetes: cordon and drain are two decisions, and conflating them moves workloads twice
• Linux: reversible and irreversible operations belong in different change windows