Ceph troubleshooting: OSDs, MONs, PGs, and recovery operations

Ceph is designed to survive hardware failures — that’s the whole point of replicated, distributed storage. But “designed to survive” and “survives without intervention” are different. Operations need to know how to diagnose, recover, and prevent recurrence.

This lesson is a deep dive into Ceph failure modes and the operational procedures for each.

The triage commands

Three commands answer 90% of Ceph questions:

# Cluster health (start here)
ceph -s
ceph health detail
ceph health

# OSD tree and state
ceph osd tree
ceph osd df tree
ceph osd stat

# Placement group state
ceph pg stat
ceph pg dump_stuck
ceph pg deep-scrub &lt;pgid&gt;   # Deep verification of a specific PG

# MON quorum
ceph mon stat
ceph mon dump

# Pool state
ceph osd pool ls detail
ceph df
ceph df detail

Every operational procedure starts with ceph -s to see the cluster state, then drills into the relevant subsystem.

OSD failure scenarios

OSD down

ceph -s shows health: HEALTH_WARN with osds down.

# Identify the failed OSD
ceph osd tree | grep down

# Find the host
ceph osd tree | grep -B1 osd.NN

# Check the host
ssh pve-XX
systemctl status ceph-osd@NN
journalctl -u ceph-osd@NN --since '1 hour ago'
# Look for I/O errors, kernel panics, OOM kills

Common causes:

Disk hardware failure — I/O errors in dmesg, SMART attributes high. Replace the disk.
Network failure — the OSD can’t reach other OSDs. Check network.
Out of memory — OSD process killed by OOM killer. Check dmesg | grep -i oom.
Ceph version mismatch — OSD won’t start with the wrong version. Check ceph versions.

Replacing an OSD safely

# 1. Mark the OSD out (Ceph stops using it)
ceph osd out osd.NN

# Wait for data to migrate (check with 'ceph -s' — pgs should
# return to active+clean)

# 2. Remove the OSD from the cluster
ceph osd purge osd.NN --yes-i-really-mean-it

# 3. Stop the OSD daemon
systemctl stop ceph-osd@NN
systemctl disable ceph-osd@NN

# 4. Wipe the disk (or replace it)
ceph-volume lvm zap /dev/sdX

# 5. Replace the disk physically if needed

# 6. Recreate the OSD on the new disk
ceph-volume lvm create --data /dev/sdX --bluestore

# 7. Verify
ceph osd tree
ceph -s

The “out” + “purge” sequence is important. If you just remove the OSD, Ceph may not have time to redistribute the data, and you’ll have under-replicated PGs.

Slow OSD

An OSD can be “up” but performing poorly — high latency, slow recovery. Symptoms:

Backfills take forever
VMs see slow disk I/O
ceph -s shows recovery: <slow progress>

# Identify slow OSDs
ceph osd perf
# Look for apply latency (commit/apply) — should be &lt;50ms typically

# Slow OSD diagnosis
ceph daemon perf osd.NN
# Shows op queue depth, latency histogram

# Check the disk
iostat -dx 1 /var/lib/ceph/osd/ceph-NN/
smartctl -a /dev/sdX
# I/O errors, high await, slow response = failing disk

Common causes:

Disk failing — replace it
Network congested — Ceph replication traffic is saturating the link
OSD process overloaded — too many PGs per OSD (target: 50–100 PGs per OSD), or competing workloads on the host
Backfill storm — many OSDs recovering at once, overloading the cluster. Throttle with ceph osd pool set <pool> noscrub during recovery.

Full OSD disk

A disk fills up. Symptoms:

ceph -s: OSD_NN is full
VM writes fail
Backfills fail

# Identify
ceph osd df | grep -E 'osd.NN|100'

# The nearfull ratio is 85%, full ratio is 95% by default
# These can be tuned:
ceph config set osd mon_osd_full_ratio 0.95
ceph config set osd mon_osd_nearfull_ratio 0.85

# Immediate relief: delete old snapshots, old data, prune RBD images
# Or expand the pool with more storage

Ceph’s protection is the mon_osd_full_ratio — when an OSD hits this threshold, the cluster stops accepting writes (a “cluster full” state). The nearfull warning at 85% gives you time to act.

MON failure scenarios

MON down

ceph -s: mons: 2/3 daemons down (or similar).

Ceph requires a majority of MONs for quorum. With 3 MONs, 1 can fail and the cluster continues. With 5 MONs, 2 can fail.

# Identify
ceph mon stat
ceph mon dump

# Check each MON's state
ceph tell mon.NN mon_status

# Restart a stuck MON
ssh mon-host-N
systemctl restart ceph-mon@$(hostname)

Replace a MON host

If a host is permanently lost:

# 1. Remove the old MON
ceph mon remove mon.NN

# 2. Deploy a new MON on a different host
ssh new-mon-host
apt install -y ceph-mon
# Get the mon keyring from another mon
scp mon-existing:/etc/ceph/ceph.mon.keyring /tmp/
ceph-mon --mkfs -i new-mon --monmap /tmp/monmap /tmp/ceph.mon.keyring

# 3. Add the new MON
ceph mon add new-mon &lt;ip&gt;

PG stuck states

ceph pg stat shows placement group states. The concerning ones:

State	Meaning
`active+clean`	Healthy (most PGs should be this)
`active+clean+remapped`	Recovery in progress
`active+clean+incomplete`	Some objects lost — needs immediate attention
`stale`	MON hasn’t heard from this PG in a while
`down`	PG’s primary OSD is down
`incomplete`	Data loss — objects not replicated
`peering`	PG is trying to reach consensus on which OSDs hold its data

A persistent incomplete or down state means data loss. The cluster will still serve reads from existing replicas but cannot guarantee durability.

# Find stuck PGs
ceph pg dump_stuck unclean
ceph pg dump_stuck stale
ceph pg dump_stuck inactive

# Investigate a specific PG
ceph pg &lt;pgid&gt; query
ceph pg &lt;pgid&gt; list_missing   # Lists objects missing from this PG

Forcing recovery when stuck

If PGs are stuck due to a buggy OSD:

# Mark the problematic OSD out
ceph osd out osd.NN

# If PGs are still stuck, force recovery
ceph pg force-recovery &lt;pgid&gt;

# Or, for stuck peering
ceph pg cancel-force-recovery &lt;pgid&gt;    # To stop a force

Force operations are dangerous — they can cause data loss if used incorrectly. Always have backups before forcing recovery.

Network partition troubleshooting

Ceph uses a separate network for replication (cluster network) and client traffic (public network). A failure in either causes symptoms:

Cluster network down — OSDs mark themselves down, peers can’t replicate, PGs go stale or incomplete
Public network down — clients can’t connect to the cluster

# Check network state from each OSD host
ceph osd pool stats
ceph osd perf

# If cluster network is down, OSDs will report each other as down
# The fix is to restore the network, not Ceph

# Verify the cluster network
ip route show
ping &lt;other-osd-host&gt;
iperf3 -c &lt;other-osd-host&gt;   # Bandwidth

Production considerations

Backfill window. Default osd_max_backfills is 1 per OSD. For faster recovery on capable hardware, raise it. For noisy networks, lower it.
Scrub schedule. Default is weekly scrub + daily deep-scrub. Don’t run scrub during peak hours; it competes with client I/O.
Recovery priority. Ceph prioritises recovery based on PG state. Incomplete PGs recover first; degraded last. The order is usually right but you can override with ceph pg force-recovery-priority.
OSD host diversity. Don’t run all OSDs of a pool on the same host — a host failure is a pool failure. CRUSH rules should spread replicas across hosts (and ideally across racks).

Common mistakes

Removing an OSD without marking it out. Data loss.
Running scrub during peak hours. User-visible latency.
Recovering on a too-full cluster. Recovery writes go to the replicas; if replicas are full, recovery fails.
Forcing recovery without understanding the cause. Force operations can paper over data loss.

Key takeaways

ceph -s first, then drill into the subsystem.
OSD down: mark out, wait for backfill, purge, recreate.
PG stuck: understand the cause before forcing recovery.
Schedule scrubs for off-peak; don’t run during peak hours.