Replacing a failed server

Why this matters in production

A host failure removes many OSDs at once, and the first decision — whether it is coming back — determines everything that follows.

Assessing the impact

ceph -s
ceph osd tree | grep -B2 -A15 ceph-03
ceph health detail

Establish:
  how many OSDs are down
  how much data they held
  whether any PG is below min_size
  whether the cluster can absorb the recovery
  what else ran on that host — monitors, managers, MDS

ceph osd df tree | grep -A15 ceph-03
ceph pg dump pgs | awk '$10 !~ /active/ {print $1, $10}' | head
ceph orch ps --hostname ceph-03

A host running a monitor as well as OSDs has affected quorum margin in addition to storage.

Deciding: restore or replace

ceph config get osd mon_osd_down_out_interval

The question is whether the OSDs return with their data intact.
If yes, restoring avoids the entire recovery.
If no, the recovery must happen and delaying it extends the exposure.

# expecting a quick return
ceph osd set noout

# not returning
ceph osd unset noout   # let the interval elapse, or mark out explicitly

Handling the recovery

# the cluster recovers automatically once the OSDs are marked out
ceph -s | grep -E 'degraded|recovery'

# verify capacity for it
ceph osd df | sort -k17 -rn | head -5

# and prioritise recovery over other work
ceph config set osd osd_mclock_profile high_recovery_ops

If any pool is at min_size, recovery takes priority over client latency until redundancy is restored.

Moving drives to another chassis

If the drives are intact and a spare chassis exists:
  move the drives, preserving their order is not required
  the OSDs are identified by their on-disk metadata, not by slot
  ceph-volume activates them on the new host

# on the new host
ceph-volume lvm list
ceph-volume lvm activate --all
ceph osd tree

# OSD_ID is one of the OSDs whose drive moved, from the `ceph osd tree` above:
OSD_ID=84

# and update CRUSH to the new host
ceph osd crush move "osd.$OSD_ID" host=ceph-03-replacement

This avoids the entire recovery, which on a twelve-drive host is substantial.

Restoring or replacing the host

# a restored host with the same name
ceph orch host ls
ceph -s

# a replacement host
ceph orch host rm ceph-03
ceph orch host add ceph-03 10.0.2.23 --labels osd
ceph osd crush move ceph-03 rack=rack1
ceph orch apply -i osd-spec.yaml

Quiz

Production discipline

Check whether a failed host’s drives are intact before accepting the recovery — an OSD’s identity lives on its device, so moving the drives to a spare chassis and activating them restores the data in minutes. Update the CRUSH position to the new hostname; that is the only movement involved.

Cross-course references

• Kubernetes: stateful workloads with portable volumes avoid rebuilding the same way
• Linux: array members carrying their own superblocks can be moved between controllers