The first minutes of an OSD failure

Why this matters in production

The first ten minutes are the window in which restoring the OSD costs nothing. What you learn in that window determines whether restoring is possible, and the down-out interval is the deadline.

The sequence

t+0     the OSD stops responding
t+20s   peers report it; the monitors mark it down
t+20s   affected PGs re-peer, becoming undersized and degraded
t+20s   client I/O continues above min_size
t+600s  the down-out interval expires; the OSD is marked out
t+600s  CRUSH remaps; backfill begins

The gap between the second and last lines is your window.

What to gather

# which OSD, and since when
ceph osd tree | grep down
ceph -s | grep osd

# the host carrying the down OSD, from the tree above:
OSD_HOST=ceph-osd-03

# is it the OSD, the host, or the network?
ssh "$OSD_HOST" systemctl status ceph-osd@13
ssh "$OSD_HOST" journalctl -u ceph-osd@13 --since '30 min ago' | tail -50

# device health
ssh "$OSD_HOST" smartctl -a /dev/sdX | grep -E 'Reallocated|Pending|SMART overall'
ssh "$OSD_HOST" dmesg -T | grep -iE 'sdX|I/O error|reset'

# does the OSD think it is running?
ssh "$OSD_HOST" journalctl -u ceph-osd@13 | grep -i 'wrongly marked me down'

The three outcomes

The daemon crashed and the device is fine. Restart it; recovery is a delta from the PG log and costs minutes.

ceph orch daemon restart osd.13
ceph -s

The device has failed. Nothing to restore. Let the interval expire or mark it out deliberately, and plan the replacement.

ceph osd out 13

The network is at fault. The OSD is fine and unreachable.

ceph osd set noout        # hold the window open
# investigate the network

Deciding within the window

ceph config get mon mon_osd_down_out_interval

If the investigation will take longer than the interval, extend the window rather than letting the transition happen by default:

ceph osd add-noout 13

Letting it expire during an investigation means a rebalance you did not choose, and a second one if the OSD turns out to be fine.

Quiz

Production discipline

Set a per-OSD noout as soon as an investigation looks likely to exceed the down-out interval; the transition happens on a timer regardless of whether you have decided. Check dmesg before the OSD log for suspected device failures — the kernel records what the device reported and the OSD does not.

Cross-course references

• Kubernetes: the node eviction grace period is the same decision window
• Linux: distinguishing device, driver, and application errors starts at the kernel log