HEALTH_ERR and the checks that produce it

Advanced⏱ ~18 minceph

What you'll learn

Enumerate the common error-severity checks
Apply the correct first response to each
Prioritise between simultaneous errors
Distinguish recoverable from data-loss situations

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

HEALTH_ERR is a short list and every entry means something is not working. Knowing the list and the first action for each is what makes the first five minutes productive.

The common error checks

Check	Means	First action
`OSD_FULL`	an OSD is at the full ratio; writes refused	free or rebalance capacity
`POOL_FULL`	a pool quota or its OSDs are full	as above
`PG_DAMAGED`	scrub found inconsistency	`ceph pg repair` after investigation
`OSD_SCRUB_ERRORS` at error level	inconsistencies confirmed	investigate before repairing
`MON_DOWN` with quorum lost	cluster not accepting updates	restore monitors
`PG_AVAILABILITY` at error level	PGs cannot serve	restore the OSDs holding them
`FS_DEGRADED`	CephFS rank without an MDS	start or fix an MDS
`MDS_ALL_DOWN`	no MDS for a rank	as above
`RECENT_CRASH`	a daemon crashed	investigate; archive when handled

Prioritising

Quorum lost — nothing else can be fixed until it is restored
Writes refused — data is not being accepted
PGs unavailable — some clients are down
Filesystem down — CephFS clients are blocked
Damaged PGs — data integrity, urgent but not blocking
Recent crash — investigate, no immediate impact

Monitor quorum comes first because most remedial commands require it.

Working the common cases

Full OSD:

ceph osd df | sort -k17 -rn | head
ceph osd reweight-by-utilization 110
ceph osd set-full-ratio 0.96      # temporary

Damaged PG:

POOL=rbd-vms
PGID=12.1a
ceph health detail | grep -A5 PG_DAMAGED
rados list-inconsistent-pg ${POOL}
rados list-inconsistent-obj ${PGID} --format=json-pretty
ceph pg repair ${PGID}

Investigate before repairing. pg repair copies from the authoritative copy, and understanding which copy is authoritative — and why the others differ — matters when the cause is a failing device that will produce more inconsistencies.

Lost quorum:

# MON_ID is the monitor's name, usually its short hostname; run this on
# that host. Substitute your own:
MON_ID=ceph-01

ceph -s                       # may hang without quorum
systemctl status ceph-mon@*
ceph daemon "mon.$MON_ID" mon_status

Recent crash:

CRASH_ID=2026-08-18T02:11:04.128Z_9f2c
ceph crash ls
ceph crash info ${CRASH_ID}
ceph crash archive ${CRASH_ID}
ceph crash archive-all

Archiving clears the check. Do it after investigating, not to silence the warning.

Quiz

Knowledge check · 4 questions

Q1. Several error checks are firing including MON_DOWN with quorum lost. What should be addressed first?

The full OSD, since refused writes affect clients immediately Monitor quorum, because most remedial commands require it The damaged PG, since data integrity outranks availability The recent crash, since it names the daemon that started this
Q2. `ceph pg repair` should be run as soon as PG_DAMAGED appears.

True False
Q3. Respond to a damaged PG.

A deep scrub has reported PG_DAMAGED with OSD_SCRUB_ERRORS at error severity on one PG. The cluster is otherwise healthy and serving normally.
Q4. Why is a daemon crash classified as an error even when the daemon restarted successfully?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Restore monitor quorum before anything else when it is among the errors; most remedial commands depend on it. Investigate the cause of a damaged PG before repairing — repair is irreversible and a failing device will simply produce the inconsistency again.

Cross-course references

Kubernetes: control plane availability precedes every other remediation for the same reason
Linux: identifying the cause of filesystem corruption before repairing is the identical discipline