Ceph day-2: OSD replacement, scrubbing, recovery

Why this matters in production

A Ceph cluster that handles routine operations well but falls apart during a failure is not production-ready. This lesson covers the day-2 procedures: how to replace an OSD, schedule scrubs, watch rebalancing, and recover from a node failure.

OSD replacement

When an OSD fails:

ceph osd tree
...
 osd.5  down  1.0  host pve-01

flowchart LR
  A[OSD 5 fails] --> B[CRUSH marks it down]
  B --> C[PGs enter degraded]
  C --> D[Recovery copies data from surviving OSDs]
  D --> E[OSD replaced, PG returns to active+clean]

The replacement procedure:

OSD=5
ceph osd out "$OSD"
ceph -s

2. Wait for the rebalance. Do not proceed while placement groups are still recovering. Removing the OSD mid-rebuild takes the affected PGs to one fewer copy than the recovery is already working with.

while ceph pg stat | grep -qE 'degraded|undersized|recovering|backfill'; do
ceph pg stat
sleep 30
done

OSD=5
pveceph osd destroy "$OSD" --cleanup 1
ceph osd tree

4. Physically replace the disk (or hot-swap if supported).

DEV=/dev/sdX

lsblk -o NAME,SIZE,SERIAL,MODEL "$DEV"
ceph-volume lvm zap "$DEV" --destroy

pveceph osd create "$DEV" --crush-device-class ssd
ceph osd tree

Scrubbing

Ceph scrubs verify data integrity by reading every object and comparing checksums. There are two kinds:

• Scrub: read metadata + data, lightweight checksum verification.
• Deep scrub: read all data, full byte-level comparison. Catches silent corruption.

Schedule both regularly:

ceph osd pool set vm-storage scrub_min_interval 86400
ceph osd pool set vm-storage deep_scrub_interval 604800

This scrubs the pool daily, with a deep scrub weekly.

Recovery and backfill

When an OSD or node fails, Ceph recovers data from surviving OSDs:

ceph -s
...
  recovery: 1234/3000 objects degraded (41%)
  recovery: 5678/3000 objects misplaced (189%)

• Degraded: fewer than size copies exist for some objects. Self-healing in progress.
• Misplaced: copies exist but on the wrong OSDs (e.g., after CRUSH rule change). Rebalancing in progress.

Recovery uses osd_recovery_max_active and osd_recovery_sleep to throttle I/O. On a busy cluster, increase recovery throughput by raising the active count; on a slow cluster, reduce it to avoid impacting workloads.

ceph config set osd osd_recovery_max_active 5
ceph config set osd osd_recovery_sleep 0

Investigating slow recovery

Common causes:

Node failure

When an entire node fails:

1. Identify the impact: which MONs, MGRs, OSDs are down.
2. Confirm the failure is real (network blip vs hardware failure).
3. Wait for automatic recovery if the failure is transient. Ceph recovers from surviving OSDs.
4. Replace the failed node if the failure is permanent.

After a node returns:

• OSDs come back up; PGs re-peer.
• If the OSDs were out long enough to trigger rebalancing, PGs may already be on other OSDs. Ceph reconciles.

ceph -s && ceph osd tree && ceph pg stat

A break/fix exercise

Production considerations

Common mistakes

• Skipping the “out + wait + stop + destroy” sequence.
• Setting noout before an OSD replacement. It suppresses exactly the rebalance you need, and the cluster sits degraded for the length of the RMA.
• Leaving noout set after a planned reboot. A genuine disk failure then never triggers a rebuild.
• Replacing multiple OSDs simultaneously (slows recovery dramatically).
• Not testing deep scrub completion.

Key takeaways

• OSD replacement follows out → wait for clean → destroy → create.
• Use pveceph osd destroy, which removes the CRUSH and auth entries together; a bare ceph osd purge can leave the ID stranded.
• noout is for planned downtime where the OSDs are coming back. It is the wrong flag for a replacement, and ceph osd stat shows what is set.
• Schedule scrubs; use deep scrubs for integrity verification.
• Throttle recovery to balance speed against load.

Knowledge check