PG states — the vocabulary you read during every incident

Intermediate⏱ ~16 minceph

What you'll learn

Interpret every common PG state
Distinguish states that serve I/O from those that do not
Prioritise between simultaneous state problems
Use PG state to direct the next diagnostic step

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

PG states are the primary diagnostic output during any Ceph incident. Reading them accurately turns a wall of text into a priority order.

The states

State	Serving?	Meaning
`active`	yes	the PG can process requests
`clean`	yes	all replicas present and agreed
`degraded`	yes	fewer copies than `size`
`undersized`	yes	acting set smaller than `size`
`remapped`	yes	acting set differs from up set
`backfilling`	yes	copying a full PG to a new OSD
`recovering`	yes	copying missed objects
`peering`	no	agreeing on state; brief
`inconsistent`	yes	scrub found disagreeing replicas
`repair`	yes	repairing an inconsistency
`scrubbing` / `deep`	yes	verification in progress
`incomplete`	no	insufficient history to proceed
`down`	no	a required OSD with needed data is unavailable
`stale`	unknown	no report from the primary recently
`inactive`	no	not serving, for any reason

Look for active first — it is the line between degraded and down

Every incident triage starts with the same question: which PGs are not active?

ceph pg stat
ceph health detail | grep -E 'inactive|incomplete|down|stale'
ceph pg dump_stuck inactive

active+undersized+degraded+backfilling looks alarming and is a cluster doing exactly what it should: serving clients on the copies it has while restoring the rest.

incomplete, down, and stale are the states that mean clients are blocked, and they are the ones to work first regardless of how many PGs are in each category.

Common combinations

active+clean                              healthy
active+undersized+degraded                a failure, still serving
active+undersized+degraded+backfilling    recovering from it
active+remapped+backfilling               planned rebalance, full redundancy
active+clean+scrubbing+deep               routine verification
active+clean+inconsistent                 scrub found a problem, still serving
peering                                   transient; persistent means investigate
incomplete                                urgent — insufficient history
down                                      urgent — needed OSD unavailable

Reading the counts

ceph pg stat
# 4096 pgs: 3900 active+clean, 190 active+undersized+degraded, 6 peering

The proportion matters as much as the state. Six PGs peering out of 4096 during a recovery is normal. Six PGs incomplete is an incident whatever the total.

Why stale is different from the other states

Every other state describes what the PG is doing. stale describes what the monitors know.

A PG is stale when the monitors have not received a status report from its primary within mon_pg_stale_timeout. The PG may be perfectly healthy — the reporting path is what has failed.

The usual causes are an OSD that has stopped without being marked down yet, a network problem between an OSD and the monitors, or an overloaded OSD too busy to report.

ceph pg dump_stuck stale

# a stale PG id from the output above:
PGID=3.1f
ceph pg "$PGID" query          # will fail if the primary is unreachable
ceph osd tree

The diagnostic move is to identify the primary of the stale PGs and check it directly. If several stale PGs share a primary, that OSD is the problem. If they share a host, that host is.

Treating stale as a data problem wastes time; it is nearly always a reporting or reachability problem.

The triage order

incomplete, down, inactive — clients blocked.
stale — establish whether it is reporting or reality.
peering persisting more than a minute or two.
inconsistent — data integrity, not availability.
undersized/degraded — redundancy, recovering on its own.
remapped/backfilling — planned movement, full redundancy.

Quiz

Knowledge check · 4 questions

Q1. Which PG state combination means clients cannot read or write that PG?

active+undersized+degraded+backfilling active+remapped+backfilling incomplete active+clean+inconsistent
Q2. A stale PG can be entirely healthy, with only the path that reports its status to the monitors having failed.

True False
Q3. ceph pg stat shows 3700 active+clean, 380 active+undersized+degraded, 12 peering, and 4 incomplete. Prioritise.

A host with 8 OSDs failed 20 minutes ago on a 96-OSD cluster. Pool size 3, min_size 2. Clients report most workloads normal but a handful of VMs are unresponsive. The team is considering marking the failed OSDs lost to clear the incomplete PGs and speed recovery.
Q4. Give the triage order for PG states during an incident and justify the top of it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Start every incident by asking which PGs are not active, because that single distinction separates blocked clients from a cluster healing itself. Read active+undersized+degraded+backfilling as working correctly rather than as an emergency. Treat stale as a reporting question and resolve it by identifying the primaries. And resist marking OSDs lost to clear incomplete — it is irreversible and usually discards data still sitting on recoverable hardware.

Cross-course references

Ceph: Part XIX (PG States) for each state in depth.
Ceph: Part XXII (PG Investigation) for the query commands.
Ceph: Part XX (PG Peering) for peering that persists.