Peering after an OSD restart

Why this matters in production

OSD restarts happen for upgrades, configuration changes, and troubleshooting. Understanding what they cost makes rolling maintenance predictable.

The sequence

t+0     OSD stops; peers notice heartbeat loss
t+20s   peers report; monitors mark it down
        → its PGs go active+undersized+degraded, still serving
t+30s   OSD restarts, loads its PGs, rejoins
        → affected PGs go peering briefly
t+35s   peering completes; recovery of missed operations begins
t+~1m   active+clean

The OSD is never marked out, because that takes 600 seconds by default. So no re-replication occurs — only the writes it missed are copied back.

ceph orch daemon restart osd.12
watch -n 5 'ceph -s | grep -E "degraded|recovering"'

Why it is cheap

The PG log covers the gap. Peering compares logs, identifies exactly which operations osd.12 missed during its 30 seconds away, and recovery copies only those.

30 seconds of writes at 100 ops/s = 3000 operations
osd_max_pg_log_entries = 10000    → comfortably covered

Compare with backfill, which would copy the OSD’s entire contents.

Rolling restarts

For a fleet-wide restart — a configuration change, a version upgrade:

ceph osd set noout
for h in ceph-01 ceph-02 ceph-03; do
  ceph orch daemon restart osd --host $h        # or per-OSD
  while ! ceph pg stat | grep -q '^.*active+clean$'; do sleep 10; done
done
ceph osd unset noout

The gating condition — waiting for active+clean between hosts — is what keeps degradation from overlapping. Restarting two hosts simultaneously on a size 3 pool with a host failure domain can take PGs below min_size.

Verifying

ceph -s
ceph pg stat
ceph osd tree | grep -w osd.12
cephadm logs --name osd.12 --since 10m | tail -20

Quiz

Production discipline

Set noout before any planned OSD or host stop, because the down-out timer cannot be set retroactively and a brief restart occasionally becomes a long one. Gate rolling restarts on active+clean between hosts rather than on a timer, since that is what prevents degradation overlapping and taking PGs below min_size. And expect busy PGs to fall out of the log window on restarts of more than a minute or two — that produces backfill and is normal rather than a fault.

Cross-course references

• Ceph: Part XCVI (Node Maintenance) for the full procedure.
• Ceph: Part LVIII (Recovery) for what recovery copies.
• Ceph: Part XCV (Maintenance Flags) for noout and relatives.