Writing Ceph alerting rules

Why this matters in production

An alert’s value is decided by whether the responder knows what to do when it fires. Threshold, duration, and annotations all serve that.

A complete rule

- alert: CephOSDLatencyOutlier
  expr: |
    topk(1, ceph_osd_apply_latency_ms) > 50
      and on() (avg(ceph_osd_apply_latency_ms) < 15)
  for: 15m
  labels:
    severity: ticket
    component: ceph
    cluster: '{{ $labels.cluster }}'
  annotations:
    summary: "{{ $labels.ceph_daemon }} apply latency {{ $value | humanize }}ms"
    description: >
      {{ $labels.ceph_daemon }} shows apply latency of
      {{ $value | humanize }}ms while the cluster average is well below.
      A single slow OSD sets the cluster p99. Check the device SMART data
      and the host before considering replacement.
    runbook_url: "https://runbooks.example.com/ceph/osd-latency-outlier"
    query: "ceph osd perf | sort -k2 -rn | head -5"

Six things: the condition, the duration, the routing labels, a summary, a description with context, and the command that starts the investigation.

Choosing thresholds and durations

The for duration should exceed the condition’s normal transient duration. Peering is sub-second, so 5 minutes is generous; a nearfull reading can fluctuate, so 30 minutes avoids noise.

Relative rather than absolute thresholds

# absolute — needs tuning per cluster and breaks after hardware changes
expr: ceph_osd_apply_latency_ms > 50

# relative — adapts to the cluster
expr: |
  ceph_osd_apply_latency_ms
    > 5 * avg(ceph_osd_apply_latency_ms)

A relative threshold survives a hardware refresh; an absolute one has to be revisited and usually is not.

Routing

route:
  routes:
    - matchers: [ component="ceph", severity="critical" ]
      receiver: pagerduty
      group_wait: 30s
    - matchers: [ component="ceph", severity="page" ]
      receiver: pagerduty
      active_time_intervals: [ business_hours ]
    - matchers: [ component="ceph", severity="ticket" ]
      receiver: ticketing
      group_interval: 1h

Testing rules

promtool check rules ceph-alerts.yml
promtool test rules ceph-alerts-test.yml

# a unit test
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'ceph_pg_total'
        values: '100+0x20'
      - series: 'ceph_pg_active'
        values: '100+0x5 95+0x15'
    alert_rule_test:
      - eval_time: 12m
        alertname: CephPGsInactive
        exp_alerts:
          - exp_labels: { severity: critical }

An alert that has never been tested is an alert whose behaviour is unknown, and the first time it is exercised is during an incident.

Quiz

Production discipline

Put the first diagnostic command or a runbook link in every alert annotation — it removes the slowest part of a 03:00 response. Prefer relative thresholds that compare each OSD against its peers, with a loose absolute backstop for the uniformly-degraded case, and unit-test rules with promtool before relying on them.

Cross-course references

• Kubernetes: alert annotations carrying kubectl commands serve the same purpose
• Linux: any alert without a stated next action costs response time