PG state metrics · Ceph · RunBook Academy

PG state metrics

Advanced⏱ ~17 mincephprometheus

What you'll learn

Map PG states to metrics
Distinguish availability from durability signals
Alert on states that do not self-resolve
Track recovery progress from PG metrics

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

PG states are the most direct expression of cluster health, and each answers a different question. Grouping them correctly is what makes the alerts meaningful.

The metrics

ceph_pg_total
ceph_pg_active
ceph_pg_clean
ceph_pg_peering
ceph_pg_degraded
ceph_pg_undersized
ceph_pg_stale
ceph_pg_inconsistent
ceph_pg_incomplete
ceph_pg_recovering
ceph_pg_backfilling
ceph_pg_backfill_toofull
ceph_pg_down

Availability versus durability

Signal	Question	States
Availability	is I/O being served?	not `active`, `peering`, `down`, `incomplete`, `stale`
Durability	are copies missing?	`degraded`, `undersized`
Integrity	do copies disagree?	`inconsistent`
Progress	is repair happening?	`recovering`, `backfilling`
Blocked progress	is repair stuck?	`backfill_toofull`

# availability — any non-zero is I/O affected
ceph_pg_total - ceph_pg_active

# durability
ceph_pg_degraded

# integrity
ceph_pg_inconsistent

Alerting on states that do not self-resolve

Peering is transient; degraded during a recovery is transient; stuck states are not:

- alert: CephPGsInactive
  expr: (ceph_pg_total - ceph_pg_active) > 0
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "{{ $value }} PGs not active — I/O to them is blocked"

- alert: CephPGsDegradedNotRecovering
  expr: ceph_pg_degraded > 0 and ceph_pg_recovering == 0 and ceph_pg_backfilling == 0
  for: 15m
  labels: { severity: page }
  annotations:
    summary: "PGs degraded with no recovery in progress"

- alert: CephPGBackfillToofull
  expr: ceph_pg_backfill_toofull > 0
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "Recovery blocked by capacity — cluster cannot restore redundancy"

- alert: CephPGInconsistent
  expr: ceph_pg_inconsistent > 0
  for: 10m
  labels: { severity: page }

The second is the most valuable: degraded PGs are normal during a recovery and abnormal when nothing is recovering.

Tracking recovery progress

# is the degraded count falling?
deriv(ceph_degraded_objects[15m]) < 0

# projected completion
ceph_degraded_objects / -deriv(ceph_degraded_objects[15m])

- alert: CephRecoveryStalled
  expr: |
    ceph_degraded_objects > 0
    and abs(deriv(ceph_degraded_objects[30m])) < 1
  for: 30m
  labels: { severity: page }
  annotations:
    summary: "Recovery has made no progress in 30 minutes"

A stalled recovery is a different problem from a slow one, and the derivative distinguishes them.

The `for` durations

Alert	`for`	Why
PGs inactive	5m	peering is normally sub-second
Degraded, not recovering	15m	allows recovery to start
Backfill toofull	5m	does not self-resolve
Inconsistent	10m	a scrub result, not transient
Recovery stalled	30m	distinguishes from slow

Quiz

Knowledge check · 4 questions

Q1. Why alert on degraded PGs with recovery at zero rather than on degraded PGs alone?

The degraded PG count is derived from PG statistics that lag, so it stays non-zero after recovery finishes Degraded PGs during an active recovery are the system working correctly; degraded with no recovery means repair is not happening The recovery metrics update on a shorter interval, so combining them gives the alert a faster reaction time Degraded PGs never require attention on their own, since min_size guarantees the data is still readable
Q2. A threshold on the degraded object count distinguishes a stalled recovery from a slow one.

True False
Q3. Build PG-state alerting.

A cluster has no PG-state alerting. The team wants to distinguish availability problems, durability problems, and stuck repairs.
Q4. Which PG states indicate an availability problem rather than a durability one?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on degraded PGs combined with zero recovering and backfilling rather than on degraded PGs alone — the first identifies a stuck repair and the second pages for every failure the cluster is handling. Use the derivative of the degraded object count to distinguish a stalled recovery from a slow one.

Cross-course references

Kubernetes: alerting on unavailable replicas with no rollout in progress is the same pattern
Linux: a RAID array degraded with no rebuild running is the identical distinction