ceph health, detail, and the mute mechanism

Intermediate⏱ ~15 mincephjq

What you'll learn

Choose the right form for the question
Use JSON output for automation
Apply and manage mutes correctly
Track health history

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The three forms answer three different questions, and mutes are the supported way to keep a known condition from masking a new one. Both are underused.

The three forms

ceph health
# HEALTH_WARN 1 osds down; Degraded data redundancy: ...

ceph health detail
# [WRN] OSD_DOWN: 1 osds down
#     osd.47 (root=default,host=ceph-osd-05) is down
# ...

ceph health detail --format json | jq '.'

Form	Answers
`ceph health`	is anything firing, and roughly what
`ceph health detail`	which specific entities
`--format json`	machine-readable, for alerting

JSON for automation

# every check, severity, and count
ceph health detail --format json | \
  jq -r '.checks | to_entries[] | "\(.value.severity) \(.key) \(.value.summary.count // "-")"'

# is a specific check firing?
ceph health detail --format json | jq -e '.checks.PG_DEGRADED' >/dev/null && echo firing

# the overall status for a simple gate
ceph health detail --format json | jq -r '.status'

Alerting should read the JSON rather than parsing the text form, which changes between releases.

Mutes

ceph health mute PG_NOT_DEEP_SCRUBBED 24h
ceph health mute OSD_DOWN 4h --sticky
ceph health unmute PG_NOT_DEEP_SCRUBBED
ceph health detail

HEALTH_OK (muted: PG_NOT_DEEP_SCRUBBED)

Property	Behaviour
Duration	required unless sticky; the mute expires
`--sticky`	persists until explicitly unmuted
Visibility	muted checks are listed in the health output
Effect on state	a muted check does not contribute
Auto-unmute	a mute clears if the condition worsens beyond what was muted

That last property matters: muting OSD_DOWN with one OSD down does not suppress it if a second goes down. The mute records the extent of the condition it was applied to.

When to mute

Appropriate: during planned maintenance where a check is expected; for a known condition awaiting a scheduled fix; to stop one persistent check masking new ones.

Inappropriate: to make a dashboard green; as an alternative to fixing something; with --sticky where a duration would do.

Health history

ceph log last 200 cluster | grep -E 'HEALTH_|cluster \['
ceph crash ls

The cluster log records health transitions, which is how you establish when a condition began and whether it has recurred.

Quiz

Knowledge check · 4 questions

Q1. You mute OSD_DOWN while one OSD is down. A second OSD then fails. What happens?

The check stays muted, since a mute covers the identifier rather than a particular count The mute clears automatically because the condition worsened beyond what was muted A second mute is created automatically for the newly-failed OSD and both expire together The cluster enters HEALTH_ERR, because two OSDs down crosses the error threshold
Q2. Alerting should parse the text output of `ceph health detail`.

True False
Q3. Keep new conditions visible during planned maintenance.

A rolling host maintenance will keep the cluster in HEALTH_WARN for two days with OSD_DOWN and PG_DEGRADED firing continuously. The team is concerned that a genuinely new problem would go unnoticed.
Q4. What makes muting safer in Ceph than blanket alert suppression elsewhere?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Consume the JSON form in automation rather than parsing text, which changes between releases. Mute expected checks during planned work so the cluster returns to HEALTH_OK and a genuinely new condition stands out — the auto-clear behaviour makes this safe in a way blanket suppression is not.

Cross-course references

Kubernetes: alert silences with expiry serve the same purpose during planned work
Linux: any maintenance-window suppression should be time-limited and visible