Getting Ceph metrics into Prometheus

Why this matters in production

Ceph’s own metrics endpoint is the authoritative source for cluster state, and getting it into an existing monitoring stack correctly determines whether the alerts you build are trustworthy.

Enabling the module

ceph mgr module enable prometheus
ceph mgr module ls | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("enabled:", [m for m in d["enabled_modules"]])'

ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
ceph config set mgr mgr/prometheus/server_port 9283
ceph config set mgr mgr/prometheus/scrape_interval 15

# Substitute your own manager host before running:
MGR_HOST=ceph-mgr-01

curl -s "http://$MGR_HOST:9283/metrics" | head -20
curl -s "http://$MGR_HOST:9283/metrics" | wc -l

Where the metrics come from

flowchart LR
  A[OSDs] --> B[Active MGR]
  C[MONs] --> B
  D[MDS] --> B
  E[RGW] --> B
  B --> F[/metrics endpoint/]
  F --> G[Prometheus]

Every metric is served by the active manager, which collects state from the daemons. That has a consequence: the endpoint moves when the manager fails over.

ceph mgr stat
ceph mgr dump | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("active:", d["active_name"], "standbys:", [s["name"] for s in d["standbys"]])'

Handling manager failover

Two approaches:

# 1. scrape all managers; standbys redirect or return nothing useful
- job_name: ceph
  static_configs:
    - targets: ['mgr-1:9283', 'mgr-2:9283', 'mgr-3:9283']
  honor_labels: true

# 2. use the standby redirect (default behaviour)
ceph config set mgr mgr/prometheus/standby_behaviour default

With the default behaviour, a standby manager redirects to the active one, so scraping any manager works. Setting it to error makes standbys return an error instead, which suits a setup where only the active should be scraped.

ceph config get mgr mgr/prometheus/standby_behaviour

Integrating with an existing Prometheus

scrape_configs:
  - job_name: 'ceph'
    scrape_interval: 15s
    honor_labels: true
    static_configs:
      - targets:
          - 'ceph-mgr-01:9283'
          - 'ceph-mgr-02:9283'
        labels:
          cluster: 'prod-ceph-01'

The cluster label matters when more than one Ceph cluster is scraped by the same Prometheus — without it, metrics from different clusters merge and every alert fires on the wrong data.

Verifying end to end

# Substitute your own manager host before running:
MGR_HOST=ceph-mgr-01

# the endpoint responds
curl -s "http://$MGR_HOST:9283/metrics" | grep -c '^ceph_'

# Prometheus is scraping it
curl -s 'http://prometheus:9090/api/v1/targets' | python3 -c '
import sys,json; d=json.load(sys.stdin)
for t in d["data"]["activeTargets"]:
    if "ceph" in t["labels"].get("job",""):
        print(t["labels"], t["health"], t.get("lastError",""))'

# a query returns data
curl -s 'http://prometheus:9090/api/v1/query?query=ceph_health_status' | \
  python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["result"])'

Quiz

Production discipline

Add a distinct cluster label when one Prometheus scrapes more than one Ceph cluster — the metric names are identical and the series otherwise merge into something misleading. List every manager as a target so failover does not create a metrics gap.

Cross-course references

• Kubernetes: multi-cluster Prometheus faces the identical label collision
• Linux: any exporter aggregating from one point makes that point critical