CephLIII · Cluster HealthCluster Health
Reading HEALTH_WARN correctly
What you'll learn
- Categorise warning checks by real urgency
- Assess scale within a warning
- Distinguish transient from persistent warnings
- Prioritise multiple simultaneous warnings
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Most of the time a cluster spends outside HEALTH_OK is spent in
HEALTH_WARN, and the range within it is enormous. Triage means asking
three questions of each warning, not looking up its name.
The three questions
1. Is client I/O affected?
ceph health detail | grep -E 'PG_AVAILABILITY|OSD_FULL|slow ops'
ceph pg dump_stuck inactiveInactive PGs, full OSDs, and sustained slow ops mean clients are affected. Everything else is background until proven otherwise.
2. What is the scale?
PG_DEGRADED: 12/25104882 objects degraded (0.00005%)
PG_DEGRADED: 8412031/25104882 objects degraded (33.5%)Same check, entirely different situations. Read the numbers, not the check name.
3. Is it moving?
ceph -s > /tmp/a; sleep 300; ceph -s > /tmp/b; diff /tmp/a /tmp/bA degraded count falling is recovery working. Unchanged is a stall.
Categorising the common warnings
Client-affecting — respond now
| Check | Means |
|---|---|
PG_AVAILABILITY | PGs not serving I/O |
SLOW_OPS | operations blocked |
POOL_FULL | writes refused |
MON_CLOCK_SKEW | consensus at risk |
Durability-affecting — respond soon
| Check | Means |
|---|---|
PG_DEGRADED | reduced redundancy; scale matters |
OSD_DOWN | capacity and redundancy reduced |
PG_BACKFILL_FULL | recovery blocked by capacity |
POOL_NEARFULL | approaching a hard limit |
Hygiene — schedule
| Check | Means |
|---|---|
PG_NOT_DEEP_SCRUBBED | corruption detection lagging |
PG_NOT_SCRUBBED | as above, shallow |
OSD_SCRUB_ERRORS | inconsistencies found — investigate |
DEVICE_HEALTH | a device is predicted to fail |
TOO_MANY_PGS | configuration to review |
OSD_SCRUB_ERRORS sits awkwardly: it is hygiene by category and warrants
prompt investigation, because it means a scrub found data that disagrees
between replicas.
Multiple simultaneous warnings
Order by client impact, then durability, then hygiene — and look for a common cause first:
ceph health detail[WRN] OSD_DOWN: 1 osds down
[WRN] PG_DEGRADED: Degraded data redundancy: 42131/25104882 objects
[WRN] PG_BACKFILL_FULL: 3 pgs backfill_toofullThree warnings, one situation: an OSD failed, its PGs degraded, and recovery is blocked by capacity on the destinations. Treating them separately means three investigations of one problem.
Quiz
Knowledge check · 4 questions
Q1. Which question should be asked first of any HEALTH_WARN?
Q2. Three simultaneous health checks usually indicate three separate problems.
Q3. Triage a cluster with several warnings.
`ceph health detail` reports: PG_NOT_DEEP_SCRUBBED on 340 PGs, PG_DEGRADED at 0.02%, OSD_DOWN with one OSD, and MON_CLOCK_SKEW on one monitor.
Q4. Why does alerting that carries only the check name discard important information?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Read the complete health detail output before acting on any
single check; one event commonly produces several and recognising the
common cause turns three investigations into one. Carry the check message,
not just the identifier, into alerts — the figures are what determine the
response.
Cross-course references
- Kubernetes: several conditions on one object frequently share a single underlying cause
- Linux: correlating alerts before investigating each is general on-call practice