Getting the most from ceph health detail

Why this matters in production

ceph health detail names the specific PGs, OSDs, and pools each check applies to, which is the difference between knowing something is wrong and knowing what to look at. Its JSON form is what makes alerting useful.

The structure

ceph health detail

HEALTH_WARN 1 osds down; Degraded data redundancy: 41213/18442104 objects degraded
[WRN] OSD_DOWN: 1 osds down
    osd.47 (root=default,host=ceph-osd-05) is down
[WRN] PG_DEGRADED: Degraded data redundancy: 41213/18442104 objects degraded (0.223%), 102 pgs degraded
    pg 7.1a is active+recovering+degraded, acting [12,47]
    pg 7.3d is active+recovering+degraded, acting [12,83]

Each check has a severity tag, an identifier, a summary, and indented detail naming the specific entities. The detail lines are where the diagnosis is.

The JSON form

ceph health detail --format json | jq '.'

# every check with its severity and summary
ceph health detail --format json | \
  jq -r '.checks | to_entries[] | "\(.value.severity)\t\(.key)\t\(.value.summary.message)"'

# the detail lines for one check
ceph health detail --format json | \
  jq -r '.checks.PG_DEGRADED.detail[].message'

# counts, for thresholds
ceph health detail --format json | \
  jq -r '.checks.PG_DEGRADED.summary.count'

The count field is what alerting should threshold on, and the detail array is what the alert should carry so the responder has the entities.

Extracting entities

# which OSDs are down
ceph health detail --format json | \
  jq -r '.checks.OSD_DOWN.detail[].message' | grep -oE 'osd\.[0-9]+'

# which PGs are degraded
ceph health detail --format json | \
  jq -r '.checks.PG_DEGRADED.detail[].message' | grep -oE 'pg [0-9a-f.]+' | awk '{print $2}'

# the OSDs those PGs have in common
ceph health detail --format json | \
  jq -r '.checks.PG_DEGRADED.detail[].message' | \
  grep -oE 'acting \[[0-9,]+\]' | tr -d 'acting[]' | tr ',' '\n' | \
  sort -n | uniq -c | sort -rn | head

That last one is the highest-value line in an incident: it counts how often each OSD appears across the affected PGs, and the one at the top is usually the cause.

Muting

ceph health mute PG_NOT_DEEP_SCRUBBED 24h
ceph health mute OSD_DOWN 2h --sticky
ceph health unmute PG_NOT_DEEP_SCRUBBED
ceph health detail | grep -i mute

Mutes are time-limited unless --sticky, and muted checks are still listed so the suppression is visible. Use them to stop a known condition masking new ones during planned work.

Quiz

Production discipline

Carry summary.message, summary.count, and a sample of the detail array into alerts so triage does not require logging in. Threshold on summary.count rather than on the detail array length — the array is capped and undercounts on large clusters.

Cross-course references

• Kubernetes: including the involved object in an event-based alert serves the same purpose
• Linux: alerts carrying the relevant identifiers rather than a category is general practice