Cluster-wide PG state: pg stat, pg dump, health detail

Why this matters in production

During an incident you need three different things at three different moments: a number (“is it getting better?”), a list (“which ones?”), and a reason (“why these?”). Reaching for pg dump when you wanted a count buries you in output; reaching for pg stat when you needed the list leaves you guessing.

The count

ceph pg stat
# 4096 pgs: 3987 active+clean, 102 active+recovering+degraded, 7 active+remapped+backfilling;
# 184 TiB data, 552 TiB used, 411 TiB / 963 TiB avail; 1.2 MiB/s rd, 44 MiB/s wr

One line, refreshable, ideal in a watch. The question it answers is “is the degraded count falling?” — and if it is not falling over several minutes, you escalate to the list.

The list

# every PG not in active+clean
ceph pg dump_stuck

# specific stuck categories
ceph pg dump_stuck inactive
ceph pg dump_stuck unclean
ceph pg dump_stuck stale
ceph pg dump_stuck undersized
ceph pg dump_stuck degraded

dump_stuck is the right first list because it already filters. The full pg dump is enormous — every PG in the cluster — and is worth it only when you need fields dump_stuck omits:

ceph pg dump --format json | \
  jq -r '.pg_map.pg_stats[] | select(.state | test("backfill_toofull")) |
         "\(.pgid)  \(.state)  up=\(.up)  acting=\(.acting)"'

The three stuck categories mean different things:

The triage

ceph health detail

HEALTH_WARN Degraded data redundancy: 41213/18442104 objects degraded (0.223%), 102 pgs degraded
[WRN] PG_DEGRADED: Degraded data redundancy: 41213/18442104 objects degraded (0.223%), 102 pgs degraded
    pg 7.1a is active+recovering+degraded, acting [12,47]
    pg 7.3d is active+recovering+degraded, acting [12,83]
    pg 7.55 is active+recovering+degraded, acting [12,91]

This is the most useful of the three during an incident because it groups by reason, not by PG, and because the acting sets are right there. In the output above, osd.12 appears in every line — that is your answer, and it took one command.

Quiz

Production discipline

Keep a watch -n 5 'ceph pg stat' running in one pane during any recovery so the trend is always visible, and use the other pane for lists and detail. The trend line is what tells you whether an intervention helped; snapshots of state do not. Save the health detail output at the start of an incident — the reason grouping is the clearest record of what the cluster looked like before you touched it.

Cross-course references

• Kubernetes: kubectl get pods --field-selector status.phase!=Running is the same filter-first instinct
• Linux: dmesg | grep -c for a count then the full log for detail is the identical escalation