The monitoring stack that ships with the design

Why this matters in production

Ceph degrades gracefully, which means it will run for weeks in a state that is quietly consuming your remaining redundancy. Alerting is what converts that silence into a ticket, and the thresholds decide whether the ticket arrives while there is still time to act on it.

The stack

ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/scrape_interval 15

ceph orch apply node-exporter --placement='*'
ceph orch apply prometheus   --placement="1 node1"
ceph orch apply alertmanager --placement="1 node1"
ceph orch apply grafana      --placement="1 node1"
ceph orch ls --service_type prometheus

# specs/prometheus.yaml — retention has to outlive the investigation
service_type: prometheus
placement:
  hosts: [node1]
spec:
  retention_time: "30d"
  retention_size: "50GB"

Fifteen days of retention loses the baseline for any incident whose question is whether this month looks like last month.

Alert on health checks, not on raw metrics

CRASH_ID=2026-08-18T02:11:04.128Z_9f2c
ceph health detail
ceph crash ls-new
ceph crash info ${CRASH_ID}

Health checks already encode Ceph’s own judgement about what is abnormal. Alerting on them gives you names that match the documentation and the mailing list, which matters at three in the morning.

Thresholds derived from time to fix

ceph osd dump | grep -E 'full_ratio'
# full_ratio 0.95  backfillfull_ratio 0.9  nearfull_ratio 0.85
ceph osd set-nearfull-ratio 0.82
ceph osd df | sort -rnk 17 | head -5

Growth 2% of capacity per week. Procurement, delivery, racking and
burn-in: 8 weeks. To have 8 weeks of runway you must be alerted at 16
percentage points below the point where you stop, so the capacity alert
fires around 70%, not at the 85% nearfull ratio.

Keeping the view when the cluster is gone

# scrape the cluster Prometheus from an external one
# - job_name: ceph-federate
#   honor_labels: true
#   metrics_path: /federate
#   params: { 'match[]': ['{job=~"ceph.*"}'] }
#   static_configs: [{ targets: ['node1:9095'] }]
curl -s http://node1:9095/federate --data-urlencode 'match[]={job=~"ceph.*"}' | head

Alertmanager notifications must leave by a path that does not traverse the cluster. A cluster-hosted Alertmanager on the host that just died sends nothing at all.

Quiz

Production discipline

Derive each threshold from how long the corresponding fix takes, and write that reasoning down next to the rule — a threshold with no stated basis gets relaxed the first time it is inconvenient. Put metrics retention and at least one Alertmanager receiver outside the cluster, because the outages worth alerting on are the ones that take the dashboard with them.

Cross-course references

• Kubernetes: alert on the condition the controller reports, not on the metric you rebuilt from scratch
• Linux: filesystem monitoring has the same lead-time arithmetic behind its thresholds