Inactive PGs — the most urgent state in Ceph

Expert⏱ ~17 minceph

What you'll learn

Identify every state that renders a PG inactive
Diagnose the cause of an inactive PG
Apply the correct remedy for each cause
Understand what marking an OSD lost does and when it is justified

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every other PG state means the cluster is serving while doing something. Inactive means it is not serving, and the users of those PGs are down.

The inactive states

State	Cause
`peering`	agreeing on history; brief normally
`incomplete`	insufficient history to establish authoritative content
`down`	an OSD holding required data is unavailable
`stale`	no report from the primary — may or may not be serving
`unknown`	monitors have no information, usually at startup

ceph pg dump_stuck inactive
ceph health detail | grep -E 'inactive|incomplete|down|stale'
ceph pg stat

Diagnosing each

peering — see the peering lesson. Query and find the blocker.

down — the PG needs an OSD that is currently unavailable:

# PGID: a stuck PG id from the dump_stuck output above
PGID=3.1f

ceph pg "$PGID" query | jq '.recovery_state[] | select(.name | contains("Down"))'
ceph pg "$PGID" query | jq '.down_osds_we_would_probe'

The remedy is to bring that OSD back. It usually holds the only copy of recent writes.

incomplete — the surviving OSDs cannot establish what the PG should contain, typically because OSDs holding recent history are gone:

# PGID: a stuck PG id from the dump_stuck output above
PGID=3.1f

ceph pg "$PGID" query | jq '.recovery_state'

stale — the monitors have not heard from the primary. Often the PG is fine and the reporting path is broken.

ceph osd lost is irreversible and tells the cluster to accept data loss

# Substitute your own value before running:
OSD_ID=17

ceph osd lost "$OSD_ID" --yes-i-really-mean-it

This declares that an OSD’s data is permanently gone, allowing PGs to proceed without it. It resolves down and incomplete PGs by accepting that whatever only that OSD held is lost.

The reasons it is nearly always wrong:

The OSD’s data usually still exists on hardware that can be recovered — a host that will not boot, a controller that failed, a drive that needs a different enclosure.
Once declared lost, the cluster proceeds with an older version of affected objects. There is no undo.
The pressure to run it is highest exactly when the diagnosis is least complete.

Before considering it: confirm the hardware is genuinely unrecoverable, attempt ceph-objectstore-tool export from the device if it can be read at all, and be able to state which objects will lose recent writes.

The response sequence

Identify the scope. How many PGs, which pools, which clients are affected.
Determine the state. Each inactive state has a different cause.
Find the missing OSDs. down_osds_we_would_probe names them.
Restore them. A host that will not boot, a drive in a failed enclosure, an OSD that will not start — each has recovery options short of accepting loss.
Only then consider ceph osd lost, with the hardware confirmed unrecoverable.

Why incomplete usually means the data still exists

incomplete means the surviving OSDs’ PG logs do not overlap enough to establish what the PG should contain. It does not mean the objects are gone.

Concretely: a PG had copies on osd.12, osd.47, and osd.83. Writes occurred while osd.47 and osd.83 were down. Then osd.12 failed. The survivors have older history and cannot know what osd.12 accepted.

The data osd.12 holds is intact on its device. The cluster simply cannot proceed without knowing what it contains.

So the productive response is to make osd.12 readable — repair its host, move its drive to another chassis, or export its PG with ceph-objectstore-tool and import it elsewhere:

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 \
    --pgid 7.3d --op export --file /tmp/pg-7.3d
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-47 \
    --op import --file /tmp/pg-7.3d

This is genuine data recovery and it works. ceph osd lost is the alternative that skips it.

After resolution

Once PGs return to active, establish why the redundancy was insufficient — size, min_size, failure domains, or overlapping maintenance — because inactive PGs mean the design did not cover what happened.

Quiz

Knowledge check · 4 questions

Q1. What does an incomplete PG state indicate about the data?

The objects have been permanently lost from every replica, which is why the PG refuses to become active again The PG holds fewer than k of its erasure-coded chunks, so the surviving shards cannot reconstruct the objects The surviving OSDs cannot establish what the PG should contain, though the data likely still exists on an unavailable OSD The PG holds objects whose replica checksums disagree, so none of the copies can be declared authoritative
Q2. ceph osd lost is a reversible operation that can be undone if the OSD is later recovered.

True False
Q3. Six PGs are incomplete after a host loss. The host will not boot but its drives appear intact. Plan recovery.

A host with 8 OSDs suffered a mainboard failure. Six PGs across two pools are incomplete; the rest recovered normally. The drives are physically undamaged and can be moved to another chassis. Two applications are down. Management is asking whether marking the OSDs lost would restore service faster.
Q4. List the states that make a PG inactive and give the first diagnostic step for each.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Work inactive PGs first in any incident, because they are the only state where clients are actually blocked. Read ceph pg <pgid> query and down_osds_we_would_probe to name the OSDs needed, then exhaust every route to making those OSDs readable — repairing a host, moving a drive, exporting a PG with ceph-objectstore-tool. Reserve ceph osd lost for hardware confirmed unrecoverable, and be able to state which objects will lose recent writes before running it.

Cross-course references

Ceph: Part CXVIII (Data Integrity Incident) for the full runbook.
Ceph: Part XXII (PG Investigation) for reading pg query.
Ceph: Part CXIV (Complete Storage Node Loss) for host-level recovery.