Verifying an evacuation is complete

Why this matters in production

Removing an OSD that still holds PGs the cluster needs is a data loss event. Verifying emptiness is what prevents it.

What complete means

An evacuated OSD:
  holds zero PGs
  reports zero used bytes beyond its overhead
  appears in no acting set
  the cluster is HEALTH_OK with no degraded or misplaced objects

All four, not just the first.

# PGs on the OSD
ID=12
ceph pg ls-by-osd ${ID} | wc -l

# its reported usage
ceph osd df | awk -v o=${ID} '$1==o {print $6, $7}'

# whether it appears in any acting set
ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys, json
osd = ${ID}
d = json.load(sys.stdin)
n = sum(1 for pg in d.get("pg_stats", []) if osd in pg.get("acting", []) or osd in pg.get("up", []))
print("appears in", n, "PG sets")'

Verifying thoroughly

for id in $(ceph osd ls-tree ceph-07); do
  printf 'osd.%-4s pgs=%-5s ' "$id" "$(ceph pg ls-by-osd $id 2>/dev/null | tail -n +2 | wc -l)"
  ceph osd df | awk -v o="$id" '$1==o {printf "used=%s\n", $7}'
done

osd.84   pgs=0     used=1.1G
osd.85   pgs=0     used=1.1G

A small residual usage is BlueStore’s own overhead and is expected; the PG count reaching zero is the signal.

ceph -s

  data:
    pgs: 4096 active+clean

Every PG active+clean with no misplaced or degraded objects means the movement finished rather than stopped.

Recognising a stall

ID=12
ceph orch osd rm status
ceph health detail | grep -i backfill
ceph pg ls-by-osd ${ID} | head

# PGID: a PG id from the ceph pg ls-by-osd output above
PGID=3.1f

# why can a specific PG not move?
ceph pg "$PGID" query | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("up:", d.get("up"), "acting:", d.get("acting"))
for s in d.get("recovery_state", [])[:1]:
    print(s.get("name"), s.get("blocked_by"))'

Confirming before removal

The gate before removing anything:
  every OSD on the node reports zero PGs
  the cluster is HEALTH_OK
  no misplaced or degraded objects remain
  ceph orch osd rm status shows the drain complete

ceph -s | grep -E 'HEALTH_OK' && \
  [ "$(for id in $(ceph osd ls-tree ceph-07); do
        ceph pg ls-by-osd $id 2>/dev/null | tail -n +2 | wc -l; done | paste -sd+ | bc)" = "0" ] && \
  echo "SAFE TO REMOVE" || echo "NOT COMPLETE"

Quiz

Production discipline

Require zero PGs on every OSD and HEALTH_OK with no misplaced objects before removing anything — the PG count alone can be zero while the cluster is still settling. Ignore residual usage on a drained OSD; BlueStore’s own structures never reach zero.

Cross-course references

• Kubernetes: a drained node with no pods is not the same as a settled cluster
• Linux: an evacuated array member must be confirmed unused before removal