The lifecycle of an OSD through its states

Intermediate⏱ ~17 minceph

What you'll learn

Trace an OSD through its full lifecycle
Identify the command driving each transition
Predict the cost of each transition
Recognise where a lifecycle has stalled

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every OSD passes through the same sequence, and knowing which transition you are performing — and what it costs — is what distinguishes a planned operation from an improvised one.

The lifecycle

graph TD
    A[device available] -->|orch apply osd| B[up / in - filling]
    B -->|backfill completes| C[up / in - steady]
    C -->|daemon fails| D[down / in]
    D -->|restart| C
    D -->|down-out interval| E[down / out]
    C -->|ceph osd out| F[up / out - draining]
    F -->|ceph osd in| C
    F -->|drain completes, stop daemon| E
    E -->|ceph osd purge| G[removed]
    E -->|start daemon, ceph osd in| C

Each transition

Transition	Command	Cost
Created → up/in	`ceph orch apply osd`	backfill in, hours
up/in → down/in	failure, or `systemctl stop`	none
down/in → up/in	`systemctl start`	recovery of the gap
down/in → down/out	timer, or `ceph osd out`	full rebalance
up/in → up/out	`ceph osd out`	full rebalance
up/out → up/in	`ceph osd in`	full rebalance back
down/out → removed	`ceph osd purge`	CRUSH weight change, further rebalance

Two costs are worth noting. down/in → up/in costs only the recovery of what changed during the absence, which is why a quick restart is cheap. And purge changes the CRUSH bucket weight, producing movement beyond what marking out already caused.

The planned retirement sequence

# 1. exclude from placement, daemon still serving
ceph osd out 13

# 2. wait for the drain
ceph -s
ceph osd safe-to-destroy 13

# 3. stop the daemon
ceph orch daemon stop osd.13

# 4. remove from the cluster
ceph osd purge 13 --yes-i-really-mean-it

ceph osd safe-to-destroy is the check that makes step 3 safe — it confirms no PG depends on this OSD’s data.

ceph osd safe-to-destroy 13
# OSD(s) 13 are safe to destroy without reducing data durability.

Where lifecycles stall

Stall	Cause
Stuck filling	backfill blocked; check `backfill_toofull`
Stuck in up/out	drain not completing; check recovery progress
down/out but not purged	forgotten cleanup, holds a CRUSH entry
Purged but device not zapped	device unavailable for reuse

# OSD map entries with no daemon
ceph osd tree | grep -E 'destroyed|down'
ceph orch ps --daemon-type osd

Comparing the OSD map against running daemons finds the entries left behind by an incomplete retirement.

Use safe-to-destroy before stopping a draining OSD

ceph osd safe-to-destroy 13

It confirms that no PG still depends on this OSD’s copy of anything. Stopping a draining OSD before it reports safe removes a copy that is still needed, taking PGs from degraded to more degraded — which is exactly what the drain was meant to avoid.

Quiz

Knowledge check · 4 questions

Q1. Why does purging an OSD cause data movement beyond what marking it out already caused?

Purging reduces the host bucket's CRUSH weight, changing placement for PGs that never involved that OSD Purging deletes the data that marking out had preserved, so every copy it held must be rebuilt from the survivors Purging resets the CRUSH map to its default hierarchy, discarding any custom bucket structure It does not — purging causes no further movement once the drain has completed and the OSD is empty
Q2. `ceph osd safe-to-destroy` should be checked before stopping a draining OSD.

True False
Q3. Retire several OSDs from a host.

Six OSDs on one host must be retired as their devices are being replaced with larger ones. The cluster has capacity and time.
Q4. Why is a quick OSD restart much cheaper than the OSD being marked out and back in?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Batch OSD retirements from the same host rather than processing them individually; the drain and the weight-change rebalance are each paid once for the group instead of once per OSD. Check safe-to-destroy before stopping any draining OSD — it is the confirmation that the drain achieved its purpose.

Cross-course references

Kubernetes: batching node removals reduces the number of rescheduling waves the same way
Linux: a documented lifecycle per component is what keeps retirements from leaving residue