OSD state metrics

Why this matters in production

up/in are independent, and the four combinations mean different things. Alerting on “OSD down” without distinguishing them produces pages for planned maintenance and silence for real problems.

The metrics

ceph_osd_up{ceph_daemon="osd.12"}    # 1 if up
ceph_osd_in{ceph_daemon="osd.12"}    # 1 if in
ceph_osd_weight
ceph_osd_metadata                    # labels: hostname, device class
ceph_osd_numpg

The four combinations

# the dangerous one: down but still in
count(ceph_osd_up == 0 and ceph_osd_in == 1)

- alert: CephOSDDownAndIn
  expr: (ceph_osd_up == 0) and (ceph_osd_in == 1)
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "{{ $labels.ceph_daemon }} is down and still in — data is under-replicated"

up=0, in=1 is the state where PGs are degraded and recovery has not yet started or completed. It is the one that warrants immediate attention.

Distinguishing expected states

# draining is expected during maintenance
ceph_osd_up == 1 and ceph_osd_in == 0

An OSD being drained is up=1, in=0 and requires no alert. An OSD that has been in that state for days, however, is an unfinished operation:

- alert: CephOSDOutTooLong
  expr: (ceph_osd_in == 0) and (ceph_osd_up == 1)
  for: 48h
  labels: { severity: ticket }
  annotations:
    summary: "{{ $labels.ceph_daemon }} has been out for 48h — unfinished drain?"

Counting rather than per-OSD alerting

On a large cluster, per-OSD alerts produce a storm during a host failure:

- alert: CephOSDsDown
  expr: count(ceph_osd_up == 0 and ceph_osd_in == 1) > 0
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "{{ $value }} OSD(s) down and in"

- alert: CephHostOSDsDown
  expr: |
    count by (hostname) (
      (ceph_osd_up == 0) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata
    ) >= 4
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "{{ $value }} OSDs down on {{ $labels.hostname }} — likely host failure"

The second is more useful than twelve individual alerts: it names the host, which is the actual failure.

Tracking transitions

changes(ceph_osd_up[1h]) > 4

An OSD transitioning repeatedly is flapping, which is a different problem from one that is simply down — usually network instability or a failing device rather than a clean failure.

Quiz

Production discipline

Page on up=0, in=1 specifically — that is the window where redundancy is reduced and recovery has not started. Aggregate by hostname using ceph_osd_metadata so a host failure produces one accurate alert rather than one per OSD.

Cross-course references

• Kubernetes: node-level alerts beat per-pod alerts for the same correlation reason
• Linux: distinguishing a failed device from a flapping one changes the remedy