Redundant Prometheus for Ceph monitoring

Advanced⏱ ~17 minprometheusalertmanager

What you'll learn

Configure redundant Prometheus instances
Understand what redundancy provides and what it does not
Handle the resulting duplicate alerts
Decide whether HA is warranted

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Prometheus is a single process with local storage, and its failure removes all Ceph visibility. Redundancy is cheap and the deduplication is handled for you.

The configuration

# both instances: identical scrape and rule configuration
global:
  external_labels:
    replica: 'prom-01'      # differs per instance
    cluster: 'prod-ceph-01'

scrape_configs:
  - job_name: 'ceph'
    static_configs:
      - targets: ['mgr-01:9283', 'mgr-02:9283', 'mgr-03:9283']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager-01:9093', 'alertmanager-02:9093']

Two instances scraping the same targets with the same rules, distinguished only by the replica external label.

Deduplication

Alertmanager receives the same alert from both replicas.
It deduplicates by the alert's label set, excluding replica labels.
One notification is sent.

# Alertmanager must not group by the replica label
route:
  group_by: ['alertname', 'cluster', 'service']

Including replica in group_by defeats the deduplication entirely and produces two notifications per alert, which is the most common misconfiguration in this setup.

What redundancy provides

Failure	Covered?
One Prometheus process dies	yes
One Prometheus host fails	yes, if on different hosts
One Prometheus runs out of disk	yes
A bad rule deployed to both	no
The Ceph exporter fails	no
Alertmanager fails	no — needs its own redundancy
A network partition isolating both	no

Redundancy at one layer covers failures at that layer only, which is why the Alertmanager and the notification path need their own treatment.

The data divergence

Each replica stores its own scrapes independently.
After a restart, one has a gap the other does not.
Queries against different replicas return different data.

This matters for dashboards: a Grafana datasource pointing at one replica shows that replica’s gaps. Options:

Approach	Effect
Load balancer across replicas	queries land on either; gaps appear intermittently
Pin dashboards to one replica	consistent, but that replica’s gaps are permanent
Thanos or Mimir	merges replicas, removing gaps

For alerting the divergence is harmless — either replica firing produces the notification. For historical dashboards it is visible.

Deciding whether HA is warranted

Situation	HA warranted?
Ceph is production-critical	yes
Alerts drive an on-call rotation	yes
Monitoring is best-effort	probably not
A single small cluster	probably not

Quiz

Knowledge check · 4 questions

Q1. What is the most common misconfiguration when running redundant Prometheus instances?

Different scrape intervals on the two replicas, which makes their rule evaluations disagree so one resolves while the other still fires Using the same rule files on both replicas, so a rule that fails to evaluate fails identically on each and the redundancy buys nothing Scraping the same targets from both replicas, which doubles the load on the active manager and can push scrape duration past the timeout Including the replica label in Alertmanager's group_by, which defeats deduplication and produces two notifications per alert
Q2. Running two Prometheus replicas makes alerting slightly more sensitive rather than more conservative.

True False
Q3. Add Prometheus redundancy for a critical Ceph cluster.

A production Ceph cluster is monitored by a single Prometheus instance. Its host failed last month and Ceph was unmonitored for six hours before anyone noticed.
Q4. Why do Prometheus replicas show different historical data?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set a distinct replica external label on each Prometheus and keep it out of Alertmanager’s group_by — including it doubles every notification. Remember that Prometheus redundancy covers only that layer; Alertmanager and the notification path need their own.

Cross-course references

Kubernetes: kube-prometheus ships this replica-and-dedup pattern by default
Linux: redundancy at one layer never covers the layers around it