Scrape configuration and reliability

Why this matters in production

A scrape that silently stops working leaves every alert built on it inactive, and nothing fires to say so. The scrape itself needs monitoring.

The configuration

scrape_configs:
  - job_name: 'ceph-prod'
    scrape_interval: 30s
    scrape_timeout: 25s
    honor_labels: true
    static_configs:
      - targets:
          - 'ceph-mgr-01.example.com:9283'
          - 'ceph-mgr-02.example.com:9283'
          - 'ceph-mgr-03.example.com:9283'
        labels:
          cluster: 'prod-ceph-01'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'ceph_rbd_.*'
        action: drop

Handling multiple clusters

  - job_name: 'ceph-dr'
    static_configs:
      - targets: ['dr-mgr-01:9283', 'dr-mgr-02:9283']
        labels:
          cluster: 'dr-ceph-01'

A separate job per cluster keeps the labelling explicit and lets each cluster have its own interval and relabelling if needed.

Monitoring the scrape

up{job=~"ceph.*"}
scrape_duration_seconds{job=~"ceph.*"}
scrape_samples_scraped{job=~"ceph.*"}

- alert: CephScrapeDown
  expr: up{job=~"ceph.*"} == 0
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "Ceph metrics scrape failing for {{ $labels.cluster }}"
    description: >
      No Ceph metrics are being collected. Every Ceph alert is inactive
      while this persists.

- alert: CephScrapeSlow
  expr: scrape_duration_seconds{job=~"ceph.*"} > 15
  for: 15m
  labels: { severity: ticket }

- alert: CephMetricsMissing
  expr: absent(ceph_health_status)
  for: 5m
  labels: { severity: page }

The absent() alert catches the case where the target is up but returning nothing useful, which up == 0 does not.

Diagnosing failures

# from the Prometheus side
curl -s 'http://prometheus:9090/api/v1/targets' | python3 -c '
import sys,json; d=json.load(sys.stdin)
for t in d["data"]["activeTargets"]:
    if "ceph" in t["labels"].get("job",""):
        print(t["labels"]["instance"], t["health"],
              round(t.get("lastScrapeDuration",0),2), t.get("lastError",""))'

# from the Ceph side
# MGR is the active manager's address, as printed by `ceph mgr services`:
MGR=192.0.2.21

ceph mgr module ls | grep -A2 prometheus
ceph mgr services
curl -s "http://$MGR:9283/metrics" | head -3

Quiz

Production discipline

Alert on up{job=~"ceph.*"} == 0 and absent(ceph_health_status) together — every Ceph alert depends on the scrape, and silence is indistinguishable from health without them. Test the full alert path with a benign triggered condition rather than assuming it works.

Cross-course references

• Kubernetes: monitoring the monitoring stack is the same dead-man requirement
• Linux: a heartbeat check is what distinguishes silence from health