The four OSD states and what each means

Intermediate⏱ ~16 minceph

What you'll learn

Interpret each of the four state combinations
Identify the normal transitions between them
Recognise an unexpected combination
Choose the correct response for each

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Two independent axes give four states, and each is a different situation. Reading both flags rather than one is what distinguishes “a disk failed” from “someone is draining this OSD”.

The four states

State	Meaning	Data on it	Response
up / in	normal operation	serving	none
up / out	running, excluded from placement	being drained	complete the drain or bring it in
down / in	absent, still in placement	inaccessible, PGs degraded	restore or let the interval expire
down / out	absent and excluded	irrelevant	remove, or restore and bring in

ceph osd stat
# 120 osds: 118 up, 117 in

ceph osd tree
ceph osd dump | grep '^osd' | awk '{print $1, $2, $3, $4}'

up / out

The OSD is running and CRUSH ignores it. Its data is being copied elsewhere, and once that completes the OSD holds nothing that is needed.

This is the state during a planned drain:

ceph osd out 13
ceph -s                    # misplaced objects falling
# once clean, the OSD is empty and can be removed

An OSD sitting in up/out indefinitely is either a drain someone forgot to finish or an OSD marked out during an incident and never brought back.

down / in

The OSD is absent and CRUSH still places on it. PGs are undersized and degraded, and nothing is moving.

This is the state immediately after a failure and before the down-out interval expires. It is also the state a noout flag holds an OSD in deliberately.

down / out

The OSD is absent and excluded. Its data has been redistributed and the cluster is repairing or repaired.

ceph osd tree | grep -E 'down'
ceph osd dump | grep '^osd' | grep 'down' | grep 'out'

An OSD here can be removed cleanly, or restored and brought back in — the latter costing a rebalance.

The normal transitions

graph LR
    A[up / in] -->|daemon fails| B[down / in]
    B -->|down-out interval| C[down / out]
    B -->|daemon restored| A
    A -->|ceph osd out| D[up / out]
    D -->|drain completes, daemon stopped| C
    D -->|ceph osd in| A
    C -->|purge| E[removed]

Unexpected combinations

Observation	Likely explanation
`up/out` for days	forgotten drain
`down/in` for hours	`noout` is set
Many `down/out`	a host or rack was lost and processed
`up/in` with zero PGs	reweight 0, or a CRUSH rule that excludes it

That last one is worth knowing: an OSD can be up, in, and holding nothing because its reweight is zero or its CRUSH rule places nothing there.

Quiz

Knowledge check · 4 questions

Q1. An OSD has been in the up/out state for three days. What does this indicate?

It has failed and the monitors have not yet recorded its daemon as absent, so it still shows as up It is being backfilled into, and CRUSH keeps an OSD out of placement until every incoming copy has finished A drain that was started and never completed, or an OSD marked out during an incident and never restored It is in maintenance mode, which cephadm applies to a host's OSDs while planned work is under way
Q2. An OSD that is up and in necessarily holds data.

True False
Q3. Audit unexpected OSD states across a cluster.

`ceph osd stat` reports `200 osds: 196 up, 193 in`. The team expects all 200 to be up and in. Nobody recalls any planned maintenance.
Q4. What does the state down/in mean about data movement?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read both the up and in counts and resolve them to per-OSD combinations; the aggregate numbers conceal whether a discrepancy is one failure or several distinct situations. Audit for OSDs persisting in up/out — forgotten drains are common and leave capacity unused.

Cross-course references

Kubernetes: node conditions and taints are similarly independent axes
Linux: distinguishing a stopped service from a disabled one is the same two-flag reading