Purpose
Ceph will tell you almost everything about itself through the manager’s Prometheus exporter, but only if something is scraping it and only if somebody is woken when it says something alarming. Clusters that fail this checklist do not fail loudly; they fail by having had a degraded PG for three weeks that nobody saw. This is the gate before the cluster is put on an on-call rotation.
Order of work
The list runs in four passes. The collection pass is prometheus: every
manager scraped, not only the currently active one, because the active manager
moves and a scrape target pinned to yesterday’s active manager goes quiet
without erroring. The presentation pass is grafana. The alerting pass is
alerting and receiver, which are deliberately separate items - a rule
that fires into a receiver nobody reads is the most common way this checklist
is passed dishonestly. The durability pass is ha and retention: the
monitoring stack must outlive the incident it is monitoring, and the history
must be long enough to show a trend.
Before you start
Have the manager hostnames, the on-call rotation the receivers should reach, and the retention window agreed with whoever pays for the storage.
Verification
Ready when every critical item is checked and one alert has been fired end-to-end to a real receiver as a test. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.
Escalation
A cluster without working alert delivery should not be accepted onto an on-call rotation. Raise that with the rotation owner.