Peering — agreeing on what the PG contains

Advanced⏱ ~16 minceph

What you'll learn

Describe what peering establishes and why it is required
Explain why peering blocks I/O for the PG
Identify what makes peering slow or stalled
Interpret peering state from pg query output

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Peering is the only routine state in which a PG does not serve. It is normally brief, and when it is not, the cluster is effectively down for the affected PGs while looking healthy in other respects.

What peering establishes

After any change to a PG’s acting set — an OSD joining, leaving, or a map update — the OSDs must agree on:

which OSD holds the most recent authoritative history,
which objects each OSD is missing,
what the PG’s current object versions are,
who is primary.

Only then can the PG serve, because serving a read before agreeing which copy is current could return stale data.

# a PG id from the dump_stuck output below:
PGID=3.1f

ceph pg "$PGID" query | jq '.state, .recovery_state[0].name'
ceph pg dump_stuck inactive

Why it blocks

A PG in peering is not active, so all client I/O to it blocks. This is correct: the alternative is serving from a state that may be superseded.

On a healthy cluster peering takes well under a second per PG, so a map change causes a brief pause that clients absorb as latency rather than as errors.

Persistent peering is an incident, not a slow operation

Peering that lasts more than a minute or two for the same PG means something is preventing agreement.

# the PG that is stuck peering, from ceph pg dump_stuck inactive
PGID=3.1f

ceph pg "$PGID" query | jq '.recovery_state'

The output names what the PG is waiting for. Common blockers:

peering_blocked_by — an OSD that must participate is unreachable.
A down OSD holding needed history — the PG cannot establish authoritative content without it.
Monitor quorum problems — no map updates, so peering cannot complete.
An overloaded OSD too busy to respond to peering messages.

The query output is the diagnostic. Reading it before acting saves guessing, and it names specific OSDs.

Reading pg query

ceph pg 7.3d query > /tmp/pg.json
jq '.state' /tmp/pg.json
jq '.recovery_state[].name' /tmp/pg.json
jq '.peer_info[].peer' /tmp/pg.json
jq '.recovery_state[] | select(.name | contains("Peering"))' /tmp/pg.json

Fields worth knowing:

Field	Meaning
`up`, `acting`	intended and current OSD sets
`recovery_state`	the state machine, most recent first
`peer_info`	what each peer reports about its copy
`blocked_by`	OSDs preventing progress
`probing_osds`	OSDs being queried for history

Why peering is memory-intensive and happens everywhere at once

Peering builds in-memory structures per PG: the missing-object set, the peer info from each participant, and the interval history.

After a map change, every OSD peers every affected PG. On a cluster with 200 PGs per OSD and a change touching most of them, that is 200 concurrent peering operations per OSD, each allocating.

The memory spike is transient and it is the peak an OSD host must be sized for. A host comfortable in steady state can exhaust memory during peering, and the OOM killer terminating an OSD produces another map change, which triggers peering again.

ceph daemon osd.12 dump_mempools | jq '.mempool.by_pool | keys'
ceph config get osd osd_memory_target

This is the mechanism behind the guidance to keep PG counts in range and to size host memory for the peering peak rather than the observed average.

The response

Confirm it is persistent, not transient — check twice, a minute apart.
ceph pg <pgid> query and read recovery_state and blocked_by.
Check the named OSDs: up, reachable, responsive.
Check monitor quorum, since no map updates means no peering completion.
Restore whatever the query names. Peering completes on its own once the blocker is gone.

Quiz

Knowledge check · 4 questions

Q1. Why does a PG in peering block client I/O?

Because the OSDs are busy copying the missing objects between themselves before they can serve Because the OSDs have not yet agreed which copy is authoritative, so a read could return stale data Because the monitors have not yet published the new map, so clients cannot compute the acting set Because peering takes an exclusive lock on the collection, so no client operation can touch the PG
Q2. Peering allocates memory per PG on every affected OSD simultaneously after a map change.

True False
Q3. Twelve PGs have been peering for fifteen minutes. The rest of the cluster is active+clean. Diagnose.

96-OSD cluster. An OSD host was rebooted 20 minutes ago and came back. Most PGs returned to active+clean quickly. Twelve remain in peering. ceph osd tree shows all OSDs up and in. Monitors are in quorum with a stable election epoch. Client I/O to the affected PGs is blocked and one application is reporting timeouts.
Q4. Name four things peering establishes and the command that shows what it is waiting for.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat peering lasting more than a minute or two as an incident rather than a slow operation, and read ceph pg <pgid> query before acting — recovery_state and blocked_by name the specific OSDs rather than leaving you to guess. Look for an OSD shared across several stuck PGs, since one unresponsive daemon that still passes heartbeats is the common cause and no health check reports it. And size host memory for the peering peak, which hits every OSD at once.

Cross-course references

Ceph: Part XX (PG Peering) for the state machine in depth.
Ceph: Part XXII (PG Investigation) for reading pg query.
Ceph: Part XI (OSD Architecture) for the memory this consumes.