Peering history — finding the authoritative copy

Advanced⏱ ~16 minceph

What you'll learn

Explain what past intervals are and why they are recorded
Describe how peering selects an authoritative OSD
Read history information from pg query
Understand why lost history produces incomplete PGs

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Peering is not about comparing data — it is about comparing history. Understanding that explains why an OSD full of correct objects can still be unable to serve, and why incomplete happens.

Past intervals

Every time a PG’s acting set changes, a new interval begins. Ceph records the sequence:

interval 1: acting [12,47,83]   epochs 41000-41180
interval 2: acting [12,47]      epochs 41181-41205   (osd.83 down)
interval 3: acting [12,47,91]   epochs 41206-

The key question during peering is: in which intervals could writes have been accepted? An interval where the acting set had at least min_size members could have accepted writes, so any OSD that was absent during it may be missing data.

ceph pg 7.3d query | jq '.info.history'
ceph pg 7.3d query | jq '.past_intervals'

Selecting the authoritative copy

Each OSD reports its PG log — the sequence of operations it has applied, with version numbers. Peering:

Collects logs from every reachable participant.
Identifies the OSD with the most recent last_update.
Computes, for every other OSD, which operations it is missing.
Designates the most-current OSD as authoritative.
Recovery then copies the missing operations.

ceph pg 7.3d query | jq '.peer_info[] | {peer, last_update, last_complete}'

Peering must consider OSDs it cannot reach

If an interval could have accepted writes and one of its participants is currently unreachable, peering cannot be certain it has the most recent data.

That is the mechanism behind incomplete: the survivors’ logs do not cover an interval in which writes may have occurred on an OSD that is now absent.

ceph pg 7.3d query | jq '.recovery_state[] | select(.name | contains("Incomplete"))'
ceph pg 7.3d query | jq '.down_osds_we_would_probe'

down_osds_we_would_probe names exactly the OSDs whose history is needed. Bringing any of them back typically resolves it immediately — which is why the productive response to incomplete is hardware recovery rather than ceph osd lost.

What the PG log holds

Each OSD keeps a bounded log of recent operations per PG:

ceph config get osd osd_max_pg_log_entries      # 10000 default
ceph config get osd osd_min_pg_log_entries
ceph pg 7.3d query | jq '.info.log_tail, .info.last_update'

If a returning OSD’s gap is covered by the log, peering identifies exactly which operations it missed and recovery copies only those. If the gap exceeds the log, the whole PG must be compared — backfill.

Why min_size affects peering and not only writes

min_size determines which intervals could have accepted writes, which determines which OSDs peering must consider authoritative.

With min_size 2 on a size 3 pool, an interval with two members could have accepted writes, so both must be consulted. An interval with one member could not, so that OSD’s absence does not block peering.

With min_size 1, every interval could have accepted writes — including intervals with a single OSD. So peering must consider every OSD that was ever alone in the acting set, and the absence of any one of them can produce incomplete.

This is a second, less obvious cost of min_size 1: it widens the set of OSDs whose loss can block peering. A pool run at min_size 1 through several failures accumulates intervals that each depend on a single OSD, and losing any of them later can leave PGs incomplete even though size 3 redundancy was nominally restored.

Reading history during an incident

# the PG you are investigating, from `ceph health detail`
PGID=3.1f

ceph pg "$PGID" query > /tmp/pg.json
jq '.info.history.same_interval_since' /tmp/pg.json
jq '.past_intervals' /tmp/pg.json
jq '.recovery_state[0]' /tmp/pg.json
jq '.down_osds_we_would_probe' /tmp/pg.json

The last is usually the actionable output: it names the OSDs to recover.

Quiz

Knowledge check · 4 questions

Q1. What does peering actually compare between OSDs?

The object data on each replica, byte by byte, before the PG is allowed to go active The PG logs and interval history, to determine which OSD holds the most recent operations The stored checksum of every object, so corruption is caught before any client can read it The free capacity of each participant, so that the emptiest OSD is designated the primary
Q2. Running a pool at min_size 1 widens the set of OSDs whose loss can later block peering.

True False
Q3. A PG is incomplete after two OSDs failed in sequence. Determine what is needed to resolve it.

PG 7.3d in a size 3 pool with min_size 2. osd.47 failed on Monday and was replaced; backfill completed. On Wednesday osd.12 failed. The PG is now incomplete. osd.83 and the replacement for osd.47 are healthy. The osd.12 host will not boot but its drives are intact.
Q4. Explain what past intervals are and why peering cares about them.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read down_osds_we_would_probe from ceph pg query as the actionable output during any incomplete incident — it names exactly the OSDs whose history is needed, and bringing any of them back usually resolves it immediately. Understand that peering compares history rather than data, which is why an OSD full of correct objects can still be unable to serve. And treat min_size 1 as widening the set of OSDs whose later loss can block peering, not only as a durability trade.

Cross-course references

Ceph: Part XIX (PG States) for incomplete and its urgency.
Ceph: Part LVIII (Recovery) for what happens after peering.
Ceph: Part CXVIII (Data Integrity Incident) for the full runbook.