Stopping the OSD daemon at the right moment

Intermediate⏱ ~15 minceph

What you'll learn

Stop an OSD daemon through the orchestrator
Verify it is safe to stop
Distinguish stopping from marking out
Handle a daemon that will not stop

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Stopping the daemon is the point of no return for that OSD’s contribution. Doing it after the drain rather than before is what keeps the whole operation non-disruptive.

Verifying first

ceph -s                          # active+clean
ceph osd safe-to-destroy 13

OSD(s) 13 are safe to destroy without reducing data durability.

Both checks. active+clean says the cluster settled; safe-to-destroy says this OSD specifically holds nothing that is still needed.

ceph osd ok-to-stop 13

ok-to-stop is the weaker check — it confirms stopping would not take PGs below min_size, which is what you want before a temporary stop rather than a removal.

Check	Answers
`ok-to-stop`	can I stop this without blocking I/O?
`safe-to-destroy`	can I remove this without losing durability?

Stopping

ceph orch daemon stop osd.13
ceph orch ps --daemon-type osd | grep osd.13

Through the orchestrator rather than systemd, so the orchestrator records the intent and does not restart it on the next reconciliation cycle.

# systemd directly would be undone
FSID=$(ceph fsid)

systemctl stop "ceph-$FSID@osd.13.service"     # the orchestrator restarts it

When the daemon will not stop

ceph orch daemon stop osd.13
ceph orch ps --daemon-type osd | grep osd.13
# still running

# check what it is doing
ceph daemon osd.13 status
ceph daemon osd.13 dump_ops_in_flight

FSID=$(ceph fsid)
journalctl -u "ceph-$FSID@osd.13" --since '10 min ago'

An OSD with outstanding operations completes them before stopping. If it does not stop within a reasonable period, its in-flight operations are the place to look.

Stopping versus marking out

	Effect
`ceph osd out 13`	excluded from placement; data moves; daemon keeps running
`ceph orch daemon stop osd.13`	daemon stops; OSD goes down; placement unchanged if still in

Stopping an OSD that is still in puts it in down/in and starts the down-out clock. That is why the drain comes first.

Quiz

Knowledge check · 4 questions

Q1. What is the difference between `ok-to-stop` and `safe-to-destroy`?

They evaluate the same condition, and ok-to-stop is retained only as an alias so older scripts keep working ok-to-stop applies to whole hosts and to daemon groups, while safe-to-destroy applies only to individual OSDs ok-to-stop asks whether a temporary absence blocks I/O; safe-to-destroy asks whether permanent removal loses durability safe-to-destroy is deprecated in favour of ok-to-stop, which now subsumes both the availability and the durability checks
Q2. Stopping an OSD daemon with systemd is equivalent to stopping it through the orchestrator.

True False
Q3. Stop an OSD safely during a removal.

osd.13 has been marked out and the rebalance completed. The cluster reports active+clean. You are about to stop the daemon before purging.
Q4. Why does stopping an OSD that is still `in` start the down-out clock?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Stop OSDs through the orchestrator so the intent is recorded and the daemon is not restarted by the next reconciliation cycle. Use ok-to-stop before a temporary stop and safe-to-destroy before a removal; they answer different questions and an OSD can pass one and fail the other.

Cross-course references

Kubernetes: deleting a pod managed by a controller has the same recreate-on-reconcile behaviour
Linux: stopping versus disabling a service is the same intent-recording distinction