Stuck peering — diagnosis and resolution

Expert⏱ ~17 minceph

What you'll learn

Recognise stuck peering and distinguish it from slow peering
Enumerate the causes and their diagnostic signatures
Apply resolutions in order of risk
Prevent recurrence

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A stuck PG serves nothing. If enough are stuck, an application is down while the cluster reports most of itself healthy.

Recognising it

ceph pg dump_stuck inactive
ceph pg stat
ceph health detail | grep -i 'stuck\|inactive\|peering'

Peering lasting more than a minute or two for the same PG is stuck. Check twice, sixty seconds apart, before concluding.

The causes and their signatures

Cause	Signature	Diagnostic
Required OSD unreachable	`blocked_by` names it	`ceph pg query`
OSD up but unresponsive	same OSD across several stuck PGs	`dump_ops_in_flight`
Monitor quorum problems	no map updates progressing	`ceph quorum_status`
Insufficient history	`incomplete` rather than `peering`	`down_osds_we_would_probe`
Network fault between OSDs	peering across a host boundary fails	interface counters
OSD out of memory	OSD restarting repeatedly	`dmesg`, OSD logs

The diagnostic sequence

# 1. scope
ceph pg dump_stuck inactive > /tmp/stuck.txt
wc -l /tmp/stuck.txt

# 2. per PG, what is it waiting for
for pg in $(awk 'NR>1 {print $1}' /tmp/stuck.txt | head -10); do
  echo "== $pg"
  ceph pg $pg query 2>/dev/null | jq -r '.recovery_state[0].name, (.blocked_by // [] | tostring)'
done

# 3. look for a common OSD
for pg in $(awk 'NR>1 {print $1}' /tmp/stuck.txt); do
  ceph pg map $pg -f json | jq -r '.up[]'
done | sort | uniq -c | sort -rn | head

# 4. check monitors
ceph quorum_status --format json-pretty | jq '.quorum_names'

Step three is usually decisive: one OSD appearing in most stuck PGs names the problem.

An OSD that is up and unresponsive is the most common cause and the least visible

Heartbeats are cheap, so an OSD can pass them while being too busy or too stuck to complete peering. Nothing marks it down, no health check fires, and it appears entirely normal in ceph osd tree.

ceph daemon osd.34 dump_ops_in_flight | jq '.num_ops'
ceph daemon osd.34 status
ceph tell osd.34 version           # times out if truly stuck
cephadm logs --name osd.34 --since 30m | tail -50

If it does not respond to ceph tell, restarting it is usually correct and low risk while the rest of the cluster is healthy:

ceph osd set noout
ceph orch daemon restart osd.34
ceph osd unset noout

Peering completes on its own once the blocker is gone.

Resolutions in order of risk

Wait sixty more seconds. Some peering is genuinely slow.
Restart the implicated OSD. Low risk, resolves the common case.
Restore an unreachable OSD or host. Whatever blocked_by or down_osds_we_would_probe names.
Fix the network or monitor quorum, if those are implicated.
ceph osd out an OSD that will not recover, allowing CRUSH to choose elsewhere.
ceph osd lost — irreversible, and only when the hardware is confirmed unrecoverable.

Why restarting the wrong OSD can extend the problem

The instinct during stuck peering is to restart OSDs until something changes. Each restart produces a map change, which triggers peering across the cluster — including the PGs already stuck, which restart their peering from the beginning.

On a cluster where peering is slow for a structural reason — very high PG counts, memory pressure, a congested network — a sequence of speculative restarts can keep the cluster in perpetual peering.

This is why step three of the diagnostic sequence matters: identify the common OSD before restarting anything, and restart only that one.

ceph pg dump_stuck inactive | awk 'NR>1 {print $1}' | while read pg; do
  ceph pg map $pg -f json | jq -r '.up[]'
done | sort | uniq -c | sort -rn | head -3

If no OSD dominates, the cause is more likely structural — monitors, network, or memory — and restarting OSDs will not help.

Preventing recurrence

After resolution, establish which cause it was and address it: memory headroom, PG counts, network reliability, or a device that should have been replaced.

Quiz

Knowledge check · 4 questions

Q1. Twenty PGs are stuck peering. What should be done before restarting any OSD?

Set noout and restart every OSD in the affected pool in sequence Identify whether one OSD appears in most of the stuck PG up sets Raise mon_lease so that peering is given more time to complete Mark the stuck PGs lost so the cluster can proceed without them
Q2. An OSD can pass heartbeats while being too stuck to complete peering.

True False
Q3. Peering is stuck across many PGs with no single OSD dominating. Monitors are in quorum. Diagnose.

96-OSD cluster, 380 PGs stuck peering following a CRUSH weight batch change. No OSD appears in more than a handful of the stuck up sets. Monitors are in quorum with a stable election epoch. dmesg on several OSD hosts shows OOM killer entries. OSDs are restarting repeatedly. PGs per OSD is 340.
Q4. List the resolutions for stuck peering in order of risk.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Identify the common OSD across stuck PGs before restarting anything — speculative restarts each trigger another cluster-wide peering round and can keep a cluster in perpetual peering. When no OSD dominates, look at monitors, network, and memory rather than daemons. Remember that an OSD passing heartbeats can still be too stuck to peer, and ceph tell osd.N version is the quick test. And work the resolutions in risk order, with ceph osd lost last and only on confirmed dead hardware.

Cross-course references

Ceph: Part XIX (PG States) for inactive states generally.
Ceph: Part XXII (PG Investigation) for reading pg query.
Ceph: Part XI (OSD Architecture) for the memory pressure case.