CephLVI · OSD FailureOSD Failure
Marking an OSD out and watching the rebalance
What you'll learn
- Mark an OSD out and predict the resulting movement
- Monitor the rebalance meaningfully
- Throttle or pause the rebalance
- Confirm the rebalance completed
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Marking out starts a process measured in hours that consumes cluster resources throughout. Knowing what to expect and how to pace it is the difference between a background operation and a visible one.
The command and its effect
ceph osd df | grep '^ *13 ' # how much will move
ceph osd out 13
ceph -s data:
pgs: 4251 active+clean
102 active+remapped+backfilling
io:
recovery: 412 MiB/s, 103 objects/sremapped means CRUSH has chosen new locations; backfilling means the
data is moving there.
Monitoring
watch -n 5 'ceph -s | grep -A5 "pgs:"'
# quantified
ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced'| Signal | Healthy | Concerning |
|---|---|---|
| Misplaced objects | falling steadily | flat for minutes |
| Recovery rate | non-zero | zero with misplaced objects remaining |
| Client latency | slightly elevated | past the SLO |
backfill_toofull | absent | any occurrence |
Estimating the duration
misplaced=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
rate=$(ceph pg stat | grep -oE '[0-9]+ objects/s' | head -1 | grep -oE '^[0-9]+')
echo "approximately $((misplaced / rate / 60)) minutes"Pacing
# slow it down
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_mclock_profile high_client_ops
# pause the movement entirely
ceph osd set norebalance
# resume
ceph osd unset norebalancenorebalance stops the misplaced-object movement while leaving genuine
degraded-PG recovery running, which is the right shape for a pause — it
defers optimisation while preserving durability work.
Confirming completion
ceph -s
# pgs: 4353 active+clean
ceph osd safe-to-destroy 13
# OSD(s) 13 are safe to destroy without reducing data durability.Both must be true before the OSD is stopped. active+clean says the
cluster is settled; safe-to-destroy says specifically that this OSD is
no longer needed.
Quiz
Knowledge check · 4 questions
Q1. During a rebalance the degraded count is falling while the misplaced count barely moves. What is happening?
Q2. `norebalance` pauses both misplaced-object backfill and degraded-PG recovery.
Q3. Pace a rebalance that is affecting clients.
An OSD was marked out and the resulting rebalance has pushed client p99 latency past the SLO. The rebalance is 20% complete with an estimated eight hours remaining.
Q4. What two conditions confirm a rebalance is complete before stopping the OSD?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Use norebalance rather than blanket throttling when a rebalance
must be paused; it defers the optimisation work and continues the
durability work. Record any paused rebalance explicitly — a cluster left
half-rebalanced is a state the next operator will find confusing.
Cross-course references
- Kubernetes: pausing a rollout while keeping healthy replicas serving is the same shape
- Linux: throttled background maintenance with a pause switch is a general pattern