Why liveness and placement are separate concerns

Intermediate⏱ ~16 minceph

What you'll learn

Explain why the two axes are independent
Describe what the separation enables
Recognise the consequences of conflating them
Apply the distinction in operational decisions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

If liveness implied placement, every brief OSD outage would trigger a full rebalance and every rebalance back. The separation is what makes routine maintenance cheap, and understanding it explains several behaviours that otherwise look arbitrary.

Two questions, two answers

Is the OSD answering? — a fact about the daemon, observed by peers, changing on the timescale of seconds.

Should data live there? — a decision about placement, made by the operator or by a timer, with a cost measured in hours.

Conflating them would mean the second question is answered every time the first changes.

What the separation enables

The down-out interval. An OSD can be absent without its data moving, so a restart, a reboot, or a brief network problem costs nothing.

ceph config get mon mon_osd_down_out_interval    # 600

Maintenance mode. A host can be taken down deliberately with noout holding its OSDs in the down/in state for as long as needed.

ceph orch host maintenance enter ceph-osd-05

Gradual draining. An OSD can be excluded from placement while still running and serving, so its data is copied off before it stops.

ceph osd out 13          # up/out: still serving reads while draining

That third case is worth noting: an up/out OSD continues to serve the data it still holds while that data is being copied elsewhere, so the drain does not reduce redundancy at any point.

The consequences of conflating them

Mistake	Consequence
Assuming down means data moved	unnecessary urgency, or missed opportunity to restore cheaply
Assuming out means the daemon stopped	stopping a daemon that was serving during a drain
Marking out to “fix” a down OSD	triggers a rebalance that was not needed
Restarting a daemon to “bring it in”	in/out is a map flag, not a daemon state

The last one appears regularly: an OSD marked out is not brought back in by restarting it. ceph osd in is the command.

Applying the distinction

Situation	Correct axis
Rebooting a host	liveness — set noout, let them go down/in
Retiring a disk	placement — out or reweight, then stop
A daemon crashed	liveness — restart it
Rebalancing away from a slow OSD	placement — reweight down
A host will be down for a week	placement — let them go out, or drain deliberately

Quiz

Knowledge check · 4 questions

Q1. An OSD was marked out during an incident and its daemon was later restarted. What state is it in?

up/in, because a restart re-registers the OSD and clears the out flag down/out, because an OSD marked out stops reporting to the monitors It will not start at all while the OSD map records it as out up/out — in and out are OSD map flags, not daemon properties
Q2. An OSD in the up/out state keeps serving the data it still holds.

True False
Q3. Plan a week-long host outage.

A host with twelve OSDs must be taken down for a week for hardware work. The cluster is size 3 with adequate capacity on the remaining hosts.
Q4. What does the separation of the two axes make possible?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Match the axis to the situation: liveness for brief absences where noout avoids unnecessary movement, placement for retirements and long-duration outages where the rebalance is the correct trade. Remember that in and out are map flags — restarting a daemon does not change them.

Cross-course references

Kubernetes: node readiness and scheduling taints are the same independent pair
Linux: a service being stopped and being disabled are similarly orthogonal