Skip to main content
RunBook Academy

CephLV · OSD StatesOSD States

in and out: the placement axis

Foundation⏱ ~16 minceph

What you'll learn

  • Define in and out precisely
  • Explain how the transition triggers data movement
  • Use out for planned removal
  • Predict the cost of marking an OSD out

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

in and out determine where data lives. Every transition on this axis moves data, which is why it is the axis to be deliberate about.

The definitions

in — CRUSH considers the OSD when computing placement.

out — CRUSH excludes it, so PGs are remapped elsewhere.

ceph osd stat
# 120 osds: 119 up, 118 in

ceph osd dump | grep '^osd'
ceph osd out 13
ceph osd in 13

The transition costs

Marking an OSD out excludes it from CRUSH, so every PG it held is recomputed and its data is copied to the new locations.

16 TB OSD marked out on a size-3 pool
  → up to 16 TB copied across the cluster
  → hours, depending on throttles and network

Marking it back in reverses the calculation and moves data again. Two transitions is two rebalances.

ceph osd out 13
ceph -s                    # watch the misplaced count
ceph osd in 13
ceph -s                    # and again

How out happens

Automatically, after the down-out interval:

ceph config get mon mon_osd_down_out_interval    # 600

Manually, for planned work:

ceph osd out 13

Gradually, through reweighting rather than a binary transition:

ceph osd reweight 13 0.5
ceph osd reweight 13 0

reweight 0 achieves the same effect as out while allowing intermediate steps, which spreads the movement over time and is gentler on a busy cluster.

Preventing the automatic transition

ceph osd set noout
ceph osd unset noout

# per OSD
ceph osd add-noout 13
ceph osd rm-noout 13

The per-OSD form is useful when one OSD is being investigated while the rest of the cluster should behave normally.

Predicting the cost

ceph osd df | grep '^ *13 '
# ID  CLASS  WEIGHT  REWEIGHT  SIZE   RAW USE  DATA  ...
# 13  hdd    16.0    1.00000   16TiB  11TiB    11TiB

RAW USE is what will be copied. Divide by the achievable recovery rate for an estimate of the duration.

Quiz

Knowledge check · 4 questions

  1. Q1. What is the cost of marking an OSD out and then back in?

  2. Q2. Marking an OSD out removes it from the CRUSH map.

  3. Q3. Investigate a failing OSD without triggering a rebalance.

    osd.13 is showing high latency and may be failing. You want to investigate for up to an hour before deciding whether to replace it. The cluster's down-out interval is at the default.

  4. Q4. How does `ceph osd reweight 13 0` differ from `ceph osd out 13`?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use noout, and the per-OSD form where only one OSD is involved, to prevent an automatic out transition while investigating; letting it happen and reversing it costs two rebalances. Reweight gradually rather than marking out abruptly when draining a busy cluster.

Cross-course references

  • Kubernetes: cordoning a node prevents new scheduling without evicting, much like noout
  • Linux: draining a member from a load balancer gradually is the same practice