The three health states and what they actually mean

Foundation⏱ ~15 minceph

What you'll learn

Define the three health states precisely
Explain why the state alone is insufficient
Map states to appropriate urgency
Read health output as a set of checks

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The health state is a summary of which checks are firing, not an assessment of how much trouble you are in. A HEALTH_WARN can mean a scrub is overdue or that a third of your data is degraded, and treating the state as the signal rather than the checks is how the second gets missed.

The states

ceph -s
ceph health
ceph health detail

State	Means
`HEALTH_OK`	no health check is currently firing
`HEALTH_WARN`	at least one check at warning severity is firing
`HEALTH_ERR`	at least one check at error severity is firing

The state is the maximum severity across all active checks. One error check makes the whole cluster HEALTH_ERR regardless of how many other things are fine.

Why the state is not the signal

ceph health detail

HEALTH_WARN 1 pgs not deep-scrubbed in time
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time

versus

HEALTH_WARN Degraded data redundancy: 8.4M/25.1M objects degraded (33.5%)
[WRN] PG_DEGRADED: Degraded data redundancy: 8412031/25104882 objects
      degraded (33.502%), 1421 pgs degraded

Both are HEALTH_WARN. One is a scheduling note and the other is a third of the cluster without redundancy.

Mapping to urgency

Check	State	Real urgency
`PG_NOT_DEEP_SCRUBBED`	WARN	low
`POOL_NEAR_FULL`	WARN	medium — rising
`PG_DEGRADED`	WARN	depends entirely on scale
`OSD_DOWN`	WARN	medium
`MON_CLOCK_SKEW`	WARN	high
`PG_AVAILABILITY`	WARN	high — I/O is blocked
`OSD_FULL`	ERR	critical
`PG_DAMAGED`	ERR	critical
`MON_DOWN` (quorum lost)	ERR	critical

Two rows are worth noting: MON_CLOCK_SKEW and PG_AVAILABILITY are warnings whose real urgency is high, which is exactly why the state alone misleads.

Reading as a set of checks

ceph health detail --format json | \
  jq -r '.checks | to_entries[] | "\(.value.severity) \(.key): \(.value.summary.message)"'

HEALTH_WARN PG_DEGRADED: Degraded data redundancy: ...
HEALTH_WARN OSD_DOWN: 1 osds down
HEALTH_WARN PG_NOT_DEEP_SCRUBBED: 4 pgs not deep-scrubbed in time

Each check has an identifier, a severity, and a message. Monitoring should alert on the checks with per-check thresholds, not on the aggregate state.

Why muted checks exist and how they change the state

ceph health mute PG_NOT_DEEP_SCRUBBED 1h
ceph health unmute PG_NOT_DEEP_SCRUBBED

A muted check is suppressed from the health output and does not contribute to the state, which allows a known and accepted condition to stop masking new ones. The mute is time-limited by default so it expires rather than persisting silently, and ceph health detail still lists muted checks separately — so the suppression is visible rather than hidden. This is the supported way to keep a cluster at HEALTH_OK during a known condition without losing the ability to notice something new.

Quiz

Knowledge check · 4 questions

Q1. A cluster reports HEALTH_WARN. What does that tell you about its severity?

Something moderately serious is wrong, since the warning level sits between OK and error on a single scale of severity Client I/O is degraded, because Ceph raises a warning only once a condition has begun to affect clients At least one warning-severity check is firing — which could be an overdue scrub or a third of the data degraded No action is required, because anything client-affecting would have been raised at error severity instead
Q2. Muting a health check hides it entirely from the health output.

True False
Q3. Improve alerting on cluster health.

A team alerts on `ceph health != HEALTH_OK`. On-call engineers receive pages for overdue scrubs at 3 am and have begun acknowledging health pages without reading them. A recent incident where a third of the data was degraded was noticed two hours late.
Q4. Which two warning-severity checks have urgency higher than their severity suggests?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on individual health check identifiers with urgency assigned per check rather than on the aggregate state; a single alert for every condition produces both fatigue and missed incidents. Use time-limited mutes for known accepted conditions so they stop masking new checks without disappearing silently.

Cross-course references

Kubernetes: alerting on individual conditions rather than a summarised status follows the same reasoning
Linux: any aggregate status indicator loses the information the responder needs