Skip to main content
RunBook Academy

CephLV · OSD StatesOSD States

up and down: the liveness axis

Foundation⏱ ~16 minceph

What you'll learn

  • Define up and down precisely
  • Describe how an OSD is marked down
  • Explain what down does not change
  • Diagnose an OSD marked down incorrectly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

up and down describe liveness and nothing else. The most common misreading is assuming that down means data has moved or that the OSD has been removed from consideration — neither is true, and the difference determines how much time you have.

The definitions

up — the OSD daemon is running and answering heartbeats.

down — peers are not receiving heartbeats and the monitors have recorded it as absent.

ceph osd stat
ceph osd tree
ceph osd tree | grep down
ID  CLASS  WEIGHT    TYPE NAME             STATUS  REWEIGHT
-1         963.00000  root default
-3         160.00000      host ceph-osd-01
 12    hdd  16.00000          osd.12           up   1.00000
 13    hdd  16.00000          osd.13         down   1.00000

Note that osd.13 is down with REWEIGHT 1.00000 — it is down and still in, so CRUSH still places PGs on it.

How an OSD is marked down

1. Peers stop receiving heartbeats
2. After osd_heartbeat_grace, peers report the failure to the monitors
3. Once mon_osd_min_down_reporters agree, the monitors mark it down
4. A new OSD map is published
ceph config get osd osd_heartbeat_grace          # 20
ceph config get mon mon_osd_min_down_reporters   # 2

Requiring several reporters prevents one OSD with a network problem from marking its peers down.

What down does not change

Effect of down
CRUSH placementunchanged
Data movementnone
PG stateundersized and degraded
Client I/Ocontinues, above min_size
Capacity accountingstill counted as in

Nothing moves. The PGs run with fewer copies and wait. Data movement begins only when the OSD is marked out, which is a separate transition.

When an OSD is marked down incorrectly

An OSD that is running but unreachable is marked down, and it notices:

journalctl -u ceph-osd@13 | grep -i 'wrongly marked me down'

That message means the network, not the OSD. Common causes are packet loss, a cluster network fault, and MTU problems.

ceph osd dump | grep osd.13
ceph daemon osd.13 status

An OSD reporting itself as running while the cluster considers it down is a network diagnosis every time.

Quiz

Knowledge check · 4 questions

  1. Q1. An OSD has been marked down. What has happened to the data it held?

  2. Q2. The monitors require several independent OSDs to report the same peer before marking it down.

  3. Q3. Respond to an OSD marked down that is actually running.

    osd.13 appears down in `ceph osd tree`. SSH to its host works, the process is running, and its log contains repeated "wrongly marked me down" entries.

  4. Q4. What is the operational significance of the window between an OSD being marked down and being marked out?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set noout early when investigating an OSD marked down that may be recoverable; the window before it is marked out is when restoring costs nothing. Treat “wrongly marked me down” as a network diagnosis without further deliberation — it is what the message means.

Cross-course references

  • Kubernetes: a node marked NotReady before pods are evicted is the same two-stage design
  • Linux: distinguishing unreachable from failed is a general distributed-systems concern