Draining before replacing

Intermediate⏱ ~16 minceph

What you'll learn

Start and monitor a drain
Determine when the drain is complete
Handle a drain that stalls
Decide whether to wait or proceed

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The drain restores full redundancy before the disk is removed. Skipping it means the replacement happens while the cluster is degraded, which is the opposite of what the procedure is for.

Starting

ceph osd out 13
ceph -s

If the OSD is already down because the device failed, the same command applies — CRUSH remaps its PGs and recovery reconstructs from the surviving copies rather than copying from this OSD.

Monitoring

watch -n 10 'ceph -s | grep -A6 "pgs:"'
ceph pg stat

4353 pgs: 4251 active+clean, 102 active+remapped+backfilling;
...
41213/18442104 objects misplaced (0.223%); 412 MiB/s, 103 objects/s recovering

The misplaced count falling is the drain progressing.

Knowing when it is over

ceph -s
# pgs: 4353 active+clean

ceph osd safe-to-destroy 13
# OSD(s) 13 are safe to destroy without reducing data durability.

Both. active+clean means no PG is degraded or misplaced anywhere; safe-to-destroy means this OSD specifically holds nothing needed.

safe-to-destroy while the drain is running reports how many PGs still depend on it, which is the most precise progress measure:

ceph osd safe-to-destroy 13
# Error EBUSY: OSD(s) 13 have 41 pgs currently mapped to them

41, then 30, then 12, then safe.

When a drain stalls

ceph -s | grep -E 'misplaced|recovery'
ceph health detail | grep -E 'backfill_toofull|PG_BACKFILL_FULL'
ceph osd df | sort -k17 -rn | head
ceph osd dump | grep flags

Cause	Signature
Destination OSDs near full	`backfill_toofull`
`norebalance` or `nobackfill` set	flags in the OSD dump
Throttles set to zero	`osd_max_backfills`
Another OSD down	recovery competing for priority

Whether to wait

Occasionally the disk must come out before the drain finishes — a hot enclosure, a technician on site with a window. The trade:

	Waiting	Proceeding
Redundancy during the swap	full	reduced
Risk of a second failure	tolerable	compounds
Time	hours	now

Proceeding is defensible when the OSD is already down and contributing nothing — pulling a dead disk changes nothing about the cluster’s state. It is not defensible when the OSD is up and draining, because pulling it removes a copy that is still serving.

Quiz

Knowledge check · 4 questions

Q1. `ceph osd safe-to-destroy 13` reports "have 41 pgs currently mapped to them". What does this mean?

An error has occurred and the check could not evaluate which PGs still depend on this OSD's own copies of data The drain is in progress and 41 PGs still depend on this OSD — the count falls to zero as it completes The OSD permanently holds 41 PGs, and the check reports its share of the cluster's placement The command is not supported for an OSD that is still up, and needs the daemon stopped first
Q2. Pulling the disk from an OSD that is already down costs the cluster nothing it has not already lost.

True False
Q3. Decide whether to proceed with a swap before the drain completes.

A technician is on site with a two-hour window. osd.13 is down because its device failed hard, and the recovery reconstructing its data from the surviving copies has four hours remaining.
Q4. What two conditions together confirm a drain is complete?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Wait for both active+clean and safe-to-destroy before pulling a disk from an OSD that is still up; the drain exists to keep redundancy intact and pulling early defeats it. Use the safe-to-destroy PG count as the progress meter — it is exact where the misplaced percentage is approximate.

Cross-course references

Kubernetes: waiting for a drain to complete before node removal is the identical requirement
Linux: waiting for a RAID rebuild before touching another member is the same discipline