A reference for the health checks you will actually see

Intermediate⏱ ~18 minceph

What you'll learn

Recognise the commonly-encountered health checks
Know the first command for each
Distinguish similar-sounding checks
Build a runbook from the reference

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

There are many possible health checks and a much shorter list that actually appears. Having the first command for each in one place is what makes the first minute of a response productive.

OSD and device

Check	Means	First command
`OSD_DOWN`	an OSD is not responding	`ceph osd tree \| grep down`
`OSD_HOST_DOWN`	every OSD on a host is down	check the host
`OSD_NEARFULL`	above `nearfull_ratio`	`ceph osd df \| sort -k17 -rn`
`OSD_BACKFILLFULL`	recovery cannot target it	as above
`OSD_FULL`	writes refused	as above, urgently
`OSD_FLAGS`	a flag such as `noout` is set	`ceph osd dump \| grep flags`
`DEVICE_HEALTH`	failure predicted	`ceph device get-health-metrics`
`DEVICE_HEALTH_TOOMANY`	too many predicted to replace safely	capacity review

Placement group

Check	Means	First command
`PG_AVAILABILITY`	PGs not serving I/O	`ceph pg dump_stuck inactive`
`PG_DEGRADED`	reduced redundancy	`ceph health detail`
`PG_BACKFILL_FULL`	recovery blocked by capacity	`ceph osd df`
`PG_DAMAGED`	inconsistency found	`rados list-inconsistent-pg`
`PG_RECOVERY_FULL`	recovery blocked	`ceph osd df`
`PG_NOT_SCRUBBED`	scrub overdue	`ceph pg dump \| grep scrub`
`PG_NOT_DEEP_SCRUBBED`	deep scrub overdue	as above
`PG_SLOW_SNAP_TRIMMING`	snapshot removal lagging	`ceph pg dump`

Monitor and manager

Check	Means	First command
`MON_DOWN`	a monitor is down	`ceph mon stat`
`MON_CLOCK_SKEW`	clocks disagree	`ceph time-sync-status`
`MON_DISK_LOW`	monitor store space	`df -h /var/lib/ceph`
`MON_DISK_CRIT`	critically low	as above, urgently
`MGR_DOWN`	no active manager	`ceph mgr stat`
`MGR_MODULE_ERROR`	a module failed	`ceph crash ls`

Pool and capacity

Check	Means	First command
`POOL_NEARFULL`	pool approaching quota or capacity	`ceph df`
`POOL_FULL`	pool at its limit	`ceph df detail`
`POOL_TOO_FEW_PGS`	below the autoscaler target	`ceph osd pool autoscale-status`
`POOL_TOO_MANY_PGS`	above it	as above
`TOO_MANY_PGS`	per-OSD PG count high	`ceph osd df`
`POOL_APP_NOT_ENABLED`	no application tag	`ceph osd pool application enable`

CephFS

Check	Means	First command
`FS_DEGRADED`	a rank has no MDS	`ceph fs status`
`MDS_ALL_DOWN`	no MDS at all	`ceph orch ps --daemon-type mds`
`MDS_INSUFFICIENT_STANDBY`	fewer standbys than wanted	`ceph fs status`
`MDS_CLIENT_RECALL`	a client is not releasing caps	`ceph tell mds.a client ls`
`MDS_CACHE_OVERSIZED`	cache above its limit	`ceph daemon mds.a cache status`
`MDS_SLOW_REQUEST`	requests blocked	`ceph daemon mds.a dump_blocked_ops`

Similar-sounding checks worth distinguishing

Pair	Difference
`OSD_NEARFULL` / `OSD_BACKFILLFULL` / `OSD_FULL`	three thresholds: warn, recovery blocked, writes refused
`PG_NOT_SCRUBBED` / `PG_NOT_DEEP_SCRUBBED`	shallow compares metadata, deep compares contents
`POOL_TOO_MANY_PGS` / `TOO_MANY_PGS`	per pool versus per OSD
`MON_DISK_LOW` / `MON_DISK_CRIT`	warning versus imminent monitor failure

Quiz

Knowledge check · 4 questions

Q1. What is the difference between POOL_TOO_MANY_PGS and TOO_MANY_PGS?

They are the same check under two names, the second kept so that alerting rules written before the rename keep matching The first is raised at warning severity and the second at error, since a cluster-wide excess blocks pool creation The first applies to replicated pools and the second to erasure-coded ones, where PG counts are computed per shard The first concerns one pool's PG count against its target; the second concerns the per-OSD PG count across the cluster
Q2. The three OSD capacity thresholds each protect a different capability.

True False
Q3. Build an incident runbook for a cluster.

A team wants to reduce the time spent recalling what each health check means during incidents. Their cluster has produced about a dozen distinct checks over two years.
Q4. What is the difference between PG_NOT_SCRUBBED and PG_NOT_DEEP_SCRUBBED?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Build a one-page runbook from the checks your cluster has actually produced, with the first command and expected response for each; the list is short and stable and a lookup beats recall at 03:00. Distinguish the similar-sounding pairs explicitly, since choosing the wrong response for the wrong threshold is a real failure mode.

Cross-course references

Kubernetes: a runbook per alert is the standard practice for the same reason
Linux: documented first-response commands per condition is general on-call discipline