Health status as a metric

Intermediate⏱ ~17 mincephprometheus

What you'll learn

Expose Ceph health checks as metrics
Alert on specific checks rather than aggregate status
Map each check to a severity and a response
Handle checks that are expected during operations

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

HEALTH_WARN fires for dozens of distinct conditions ranging from trivial to urgent. Alerting on the aggregate status trains the team to ignore it.

The metrics

ceph_health_status               # 0 = OK, 1 = WARN, 2 = ERR
ceph_health_detail{name="..."}   # 1 when that specific check is firing

# Substitute the active manager's address before running:
MGR_HOST=192.0.2.11

curl -s "http://$MGR_HOST:9283/metrics" | grep '^ceph_health_detail'

ceph_health_detail{name="OSD_NEARFULL",severity="HEALTH_WARN"} 1.0
ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED",severity="HEALTH_WARN"} 1.0

The per-check metric is what makes specific alerting possible.

Mapping checks to severity

Check	Severity	Response
`OSD_FULL`	page	writes blocked
`PG_AVAILABILITY`	page	I/O blocked
`MON_DOWN`	page	quorum at risk
`PG_DAMAGED`	page during hours	data integrity
`OSD_DOWN`	page during hours	durability reduced
`PG_BACKFILL_FULL`	page	recovery stalled
`OSD_NEARFULL`	ticket	capacity planning
`PG_NOT_DEEP_SCRUBBED`	ticket	scrub schedule
`POOL_APP_NOT_ENABLED`	ticket	configuration
`OSDMAP_FLAGS`	ticket	a flag is set
`TOO_MANY_PGS`	ticket	configuration
`RECENT_CRASH`	ticket	investigate

- alert: CephOSDFull
  expr: ceph_health_detail{name="OSD_FULL"} == 1
  for: 1m
  labels: { severity: critical }

- alert: CephNearfull
  expr: ceph_health_detail{name="OSD_NEARFULL"} == 1
  for: 30m
  labels: { severity: ticket }

The for duration differs by check: a full OSD warrants a minute, a nearfull one does not need to fire on a transient reading.

Handling expected checks

During planned work, certain checks are expected:

maintenance: OSD_DOWN, PG_DEGRADED, OSDMAP_FLAGS (noout)
expansion:   OBJECT_MISPLACED, PG_BACKFILLING
upgrade:     OSD_DOWN, MON_DOWN briefly

Silencing them for the window is correct; silencing them indefinitely is not:

# a silence with an expiry, not a permanent inhibition
amtool silence add alertname=CephOSDDown --duration=4h \
  --comment="planned maintenance, ticket OPS-1234"

# or an inhibition rule tied to a maintenance indicator
- source_matchers: [ maintenance_active="true" ]
  target_matchers: [ alertname=~"CephOSD.*" ]

The aggregate status

ceph_health_status > 1        # HEALTH_ERR only

The aggregate remains useful as a backstop for conditions no specific alert covers, at a low severity — but it should not be the primary signal.

Quiz

Knowledge check · 4 questions

Q1. Why is alerting on `ceph_health_status` alone inadequate?

The value is computed only when a health check changes, so it can lag the cluster's real state by minutes It updates only at the mgr's own scrape interval, which is longer than most alert evaluation periods HEALTH_WARN covers conditions from trivial to urgent, so one alert either fires constantly or misses the urgent cases It reflects only the monitor-side checks, so OSD and PG conditions never raise it at all
Q2. A silence without an expiry is acceptable for alerts suppressed during maintenance.

True False
Q3. Rebuild Ceph alerting.

A team has one alert on ceph_health_status != 0. It fires several times a week for PG_NOT_DEEP_SCRUBBED and POOL_APP_NOT_ENABLED, and is routinely acknowledged without reading.
Q4. Why is an inhibition rule tied to a maintenance indicator better than a manual silence?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on individual checks via ceph_health_detail with per-check severities and for durations, keeping ceph_health_status only as a low-severity backstop. Use expiring silences or self-clearing inhibitions during maintenance — the suppressed alerts are exactly the ones covering what maintenance can leave behind.

Cross-course references

Kubernetes: alerting on aggregate cluster status has the identical problem
Linux: a single “system unhealthy” alert always degrades into noise