When a PG has no surviving copy

Advanced⏱ ~18 mincephceph-objectstore-toolcephadm

What you'll learn

Distinguish down from incomplete from unfound
Identify which OSDs the PG needs
Extract a PG from a dead OSD store
Sequence the destructive options correctly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A PG marked down usually still has its objects somewhere, and the commands that make the warning go away are the ones that make that untrue.

What down means

ceph pg dump_stuck inactive
ceph health detail | grep -E 'PG_AVAILABILITY|PG_DAMAGED'

State	Cause	Data
`down`	an OSD holding required history is unavailable	probably intact
`incomplete`	the surviving copies cannot form a complete history	possibly intact
`unfound` objects	copies are known to have existed and cannot be located	intact only if a down OSD returns
`stale`	no OSD has reported on this PG recently	unknown

ceph pg 3.1f query | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("state:", d.get("state"))
for s in d.get("recovery_state", []):
    print("-", s.get("name"))
    for k in ("probing_osds","down_osds_we_would_probe",
              "peering_blocked_by","blocked"):
        if s.get(k):
            print("   %-26s %s" % (k, s[k]))'

down_osds_we_would_probe  [23, 41]

The order of attempts

Attempt	Cost	Reversible
Return one of the probed OSDs	none	yes
Move the disk to another chassis and start the OSD there	hardware time	yes
Export the PG from the dead store and import it elsewhere	expert time	yes
`ceph osd lost`	may create unfound objects	no
`ceph pg mark_unfound_lost`	discards the named objects	no
`ceph osd force-create-pg`	discards the entire PG	no

Extracting a PG from a dead store

cephadm unit --name osd.23 stop
cephadm shell --name osd.23 -- ceph-objectstore-tool \
  --data-path /var/lib/ceph/osd/ceph-23 --op list-pgs

cephadm shell --name osd.23 -- ceph-objectstore-tool \
  --data-path /var/lib/ceph/osd/ceph-23 \
  --pgid 3.1f --op export --file /var/lib/ceph/osd/ceph-23/3.1f.export

# import into an OSD that does not already hold this PG, while it is stopped
cephadm unit --name osd.77 stop
cephadm shell --name osd.77 -- ceph-objectstore-tool \
  --data-path /var/lib/ceph/osd/ceph-77 --op import --file /mnt/rescue/3.1f.export
cephadm unit --name osd.77 start

The export is a read of the local store. It does not require the cluster
to accept the OSD, only that the device is readable and the OSD process
is stopped.

The last resorts

ceph osd lost 23 --yes-i-really-mean-it
ceph pg 3.1f list_unfound | head

ceph osd force-create-pg 3.1f --yes-i-really-mean-it

Why a PG refuses to serve copies it demonstrably holds

Peering does not compare object contents; it compares interval history. Each PG records the epochs during which it was able to serve, and a surviving OSD can only prove its copy is current if the OSDs that were acting during any later interval are present to confirm nothing happened in that window. When those OSDs are unavailable, the survivor has an unprovable copy: it may be complete, or it may be missing every write taken during an interval it was not part of. Serving it would return stale data and silently discard acknowledged writes, so Ceph marks the PG down and names the OSDs it needs to hear from. That is why down is a statement about history rather than about data, and why bringing back a probed OSD for even a few minutes often resolves it completely.

Quiz

Knowledge check · 4 questions

Q1. What does `down_osds_we_would_probe` in a PG query tell you?

Which OSDs hold the only remaining copies of the data, so the objects are unrecoverable without them Which OSDs have already been marked lost, and whose objects peering has written off Which OSDs are being scrubbed, and therefore cannot answer a peering request yet Which unavailable OSDs peering needs to hear from before it can prove a surviving copy is current
Q2. A PG can be marked down even though every one of its objects is intact on a surviving OSD.

True False
Q3. Recover a PG with no available copy.

Three OSDs across three hosts failed together. PG 3.1f is down and its query lists osd.23 and osd.41 under down_osds_we_would_probe. Both hosts failed on a power event; the disks are believed intact.
Q4. What does `ceph osd force-create-pg` do to the data in that PG?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read down_osds_we_would_probe before anything else — it converts a lost PG into a named hardware task. Confirm in writing that every probed OSD is physically unrecoverable before ceph osd lost or force-create-pg, since both return the cluster to HEALTH_OK with no record of what was discarded.

Cross-course references

Kubernetes: a consensus store refuses to elect rather than serve an unprovable log
Linux: a filesystem that will not mount is often intact and missing only its journal context