Operating the metrics exporter

Why this matters in production

On a large cluster the exporter produces a large response and takes real time to generate it. Left at defaults it can affect manager responsiveness and produce scrape timeouts.

The cost

# Substitute the active manager's address before running:
MGR_HOST=192.0.2.11

time curl -s "http://$MGR_HOST:9283/metrics" > /dev/null
curl -s "http://$MGR_HOST:9283/metrics" | wc -l
curl -s "http://$MGR_HOST:9283/metrics" | awk -F'{' '{print $1}' | sort | uniq -c | sort -rn | head

  76800 ceph_pg_
  38400 ceph_osd_
   9600 ceph_pool_

On a 500-OSD cluster with 32,768 PGs the response can exceed a hundred thousand lines and take several seconds to generate.

Configuring for scale

ceph config set mgr mgr/prometheus/scrape_interval 30
ceph config set mgr mgr/prometheus/cache true
ceph config set mgr mgr/prometheus/stale_cache_strategy fail
ceph config set mgr mgr/prometheus/rbd_stats_pools ''

The cache setting matters most: with it enabled the exporter generates the response on its own schedule and serves it to any number of scrapers, rather than regenerating per scrape.

# the exporter's own scrape interval must be <= Prometheus's
ceph config get mgr mgr/prometheus/scrape_interval

If Prometheus scrapes more often than the exporter refreshes, it receives the same data repeatedly — harmless but pointless. If it scrapes less often, the cache goes stale and the strategy determines what happens.

Controlling cardinality

# per-image RBD statistics are high cardinality
ceph config get mgr mgr/prometheus/rbd_stats_pools
ceph config set mgr mgr/prometheus/rbd_stats_pools 'rbd-vms'

Enabling RBD per-image statistics on a pool with ten thousand images adds tens of thousands of series. It is valuable for a pool where per-image visibility is needed and expensive everywhere else.

# check the resulting cardinality
MGR_HOST=192.0.2.11

curl -s "http://$MGR_HOST:9283/metrics" | grep -c '^ceph_rbd'

Diagnosing problems

# what Prometheus sees
curl -s 'http://prometheus:9090/api/v1/targets' | python3 -c '
import sys,json; d=json.load(sys.stdin)
for t in d["data"]["activeTargets"]:
    if "ceph" in str(t["labels"]):
        print(t["labels"].get("instance"), t["health"], t.get("lastScrapeDuration"))'

# raise the scrape timeout if generation is genuinely slow
- job_name: ceph
  scrape_interval: 30s
  scrape_timeout: 25s

Quiz

Production discipline

Enable the exporter cache on any cluster where generating the metrics response takes measurable time — it decouples manager load from the number of scrapers entirely. Set stale_cache_strategy to fail so stale values are never recorded as current and alerts evaluate on real data.

Cross-course references

• Kubernetes: kube-state-metrics faces the same cardinality and generation-cost issues
• Linux: any exporter that enumerates many objects needs the same scale consideration