What HEALTH_OK does not tell you

Why this matters in production

HEALTH_OK means no check is firing. Every serious cluster problem passes through a period where nothing is firing yet, and several important risks never fire a check at all.

What it does not cover

Every row is a real risk and none produces a warning.

The gaps that matter most

Capacity trajectory. HEALTH_OK at 78% full and rising 2% a week is three weeks from a warning and five from an incident.

ceph df
ceph osd df | awk '{print $17}' | sort -n | tail -3

Latency outliers. An OSD at ten times the median makes the cluster feel broken and fires no check.

ceph osd perf | sort -k2 -n | tail -5

Configuration risk.

ceph osd pool ls detail | grep -E 'min_size 1|size 2'
ceph osd crush rule dump | jq -r '.[] | "\(.rule_name) \(.steps[1].type // "?")"'
ceph osd tree | grep -c host

Redundancy that has never been tested. A cluster that has never lost a host does not know whether it survives losing one.

Building the gap coverage

# capacity trajectory — record and trend
ceph df --format json | jq -r '.stats.total_used_raw_ratio'

# latency distribution
ceph osd perf --format json | \
  jq -r '.osdstats.osd_perf_infos[] | "\(.id) \(.perf_stats.commit_latency_ms)"'

# configuration audit
ceph osd pool ls detail
ceph config dump | grep -v '^WHO'
ceph auth ls | grep -B2 'allow \*'

These belong on a schedule, not in the health output — Ceph does not consider them health questions and it is right not to.

Quiz

Production discipline

Treat HEALTH_OK as one input rather than as an answer; capacity trajectory, latency distribution, and configuration risk are all invisible to it and all real. Schedule the configuration audit rather than relying on periodic reviews, since the conditions it covers do not announce themselves.

Cross-course references

• Kubernetes: all pods Running says nothing about resource headroom or configuration risk
• Linux: a green status page and a healthy system are related but distinct claims