Ceph troubleshooting scenarios

Why this matters in production

Ceph incidents tend to look the same from the dashboard but have very different causes. This lesson walks through the most common patterns.

Pattern 1: HEALTH_WARN — nearfull on an OSD

health: HEALTH_WARN
  1 nearfull OSD(s)

Cause: an OSD is using more than nearfull_ratio (default 0.85) of its capacity.

Diagnostic:

ceph osd df | head -20

Fixes:

• Add capacity (replace the OSD’s disk with a larger one).
• Rebalance data by adding more OSDs.
• Delete unused snapshots or data.

Pattern 2: PGs stuck in degraded

health: HEALTH_WARN
  12 active+degraded+undersized+remapped pgs

Cause: not enough OSDs to satisfy the pool’s CRUSH rule. Common after a node failure when replicas cannot be placed.

Diagnostic:

ceph pg dump_stuck stale && ceph pg dump_stuck unclean && ceph pg dump_stuck inactive

Fix: restore the missing OSDs. Replace the hardware, fix the network, bring the node back. That is the entire correct response, and the reason it is worth stating flatly is that the tempting alternative is worse than the problem.

Pattern 3: OSD flapping

health: HEALTH_WARN
  1 OSD(s) experiencing slow operations

An OSD is starting and stopping repeatedly. Common causes:

• Network instability.
• Disk failure (intermittent).
• Software bug in the OSD process.

Diagnostic:

ceph osd tree && ceph osd metadata 5 | grep -E 'host|addr'

ceph daemon osd.5 dump_ops_in_flight 2>/dev/null || journalctl -u ceph-osd@5 --since '10 minutes ago'

Fix: identify the root cause (hardware replacement, network fix, software update).

Pattern 4: Slow ops cluster-wide

health: HEALTH_WARN
  47 slow ops, oldest in 230s

Something in the cluster is bottlenecked. Diagnostic steps (see the Ceph performance lesson for detail):

1. ceph osd perf — find slow OSDs.
2. ceph daemon osd.N perf dump — check op_queue vs op_process latency.
3. Check NIC counters.
4. Check for stuck PGs (recovery in progress).

Fix: address the bottleneck. Common remedies: throttle recovery, replace slow disks, add network capacity.

Pattern 5: Full cluster (no recovery possible)

If the cluster reaches full_ratio (default 0.95), Ceph blocks all writes to prevent data loss.

Diagnostic:

ceph df && ceph osd df tree | sort -k5 -n -r | head

Fix:

1. Increase full_ratio and nearfull_ratio temporarily to give yourself room to investigate. This is a stopgap, not a fix.
2. Delete unused snapshots.
3. Resize the cluster (add OSDs).
4. Restore the ratios to defaults once stable.

# what are they now?
ceph osd dump | grep -E 'full_ratio|nearfull_ratio'

ceph osd set-full-ratio 0.97
ceph osd set-nearfull-ratio 0.90
ceph osd set-backfillfull-ratio 0.92

Pattern 6: Cascading failure after node loss

When a node fails and recovery begins, surviving OSDs see increased load. If they were already at 70-80 % utilisation, the additional load can:

• Push them to nearfull.
• Slow response time.
• Trip Ceph’s slow-op detection.
• Cause recovery to stall.

This is the most dangerous Ceph failure mode. Prevention:

• Keep utilisation well below nearfull_ratio.
• Maintain N+2 capacity headroom (not just N+1).
• Test node failures on a non-production cluster.

A break/fix exercise

Production considerations

Common mistakes

• Raising full_ratio instead of adding capacity.
• Lowering size or min_size to clear a degraded warning. It relabels the problem, discards the copy Ceph was rebuilding, and the min_size version is almost never reverted afterwards.
• Using ceph osd set full_ratio 0.97. That is not the command: ceph osd set takes flags such as noout. The threshold commands are ceph osd set-full-ratio, set-nearfull-ratio and set-backfillfull-ratio.
• Restarting OSDs in random order during a cascade.
• Trusting HEALTH_OK without checking recovery progress.
• Not having replacement OSDs on hand.

Key takeaways

• nearfull → add capacity.
• Degraded PGs → restore missing OSDs. Never lower size or min_size to make the warning go away.
• Slow ops → find the bottleneck (network/disk/load).
• Full cluster → raise the ratios by a small amount with ceph osd set-full-ratio, and restore them in the same change record.

Knowledge check