Scaling the monitoring stack with the cluster

Advanced⏱ ~17 minprometheusceph

What you'll learn

Estimate monitoring resource requirements from cluster size
Plan retention against storage
Recognise the symptoms of an undersized stack
Scale the components appropriately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Monitoring capacity is planned once and then forgotten, while the cluster it monitors triples. The failure is a monitoring outage during the period of highest need.

Estimating requirements

series ≈ (OSDs × ~40) + (pools × ~20) + (PGs-related, fixed) + (images × ~10 if RBD stats on)

500 OSDs × 40      = 20,000
20 pools × 20      =    400
fixed families     =  ~1,000
                     ------
                     ~21,400 series

Prometheus memory ≈ series × ~3 KB (with default settings)
                  ≈ 21,400 × 3 KB ≈ 64 MB for the head block

The head block is only part of it — query load, rule evaluation, and compaction all add. A practical figure is several gigabytes for a cluster of this size, and it scales with series count.

# actual, from Prometheus
curl -s http://prometheus:9090/api/v1/status/tsdb | python3 -c '
import sys,json; d=json.load(sys.stdin)["data"]
print("series:", d["headStats"]["numSeries"])
print("chunks:", d["headStats"]["chunkCount"])'

Retention against storage

bytes ≈ series × samples_per_second × retention_seconds × ~1.7 bytes

21,400 series at 15s interval, 90 days:
  21,400 × (1/15) × (90 × 86400) × 1.7 ≈ 19 GB

--storage.tsdb.retention.time=90d
--storage.tsdb.retention.size=50GB

Setting both means the size limit acts as a backstop if the series count grows unexpectedly, which is what prevents a cardinality change filling the disk.

Symptoms of an undersized stack

Symptom	Cause
Prometheus OOM-killed	series count exceeded memory
Queries timing out	too many series per query, or slow storage
Dashboard load times rising	same
Rule evaluation lagging	`prometheus_rule_group_last_duration_seconds` rising
Gaps in data	scrape failures or restarts
Disk full	retention not bounded by size

prometheus_rule_group_last_duration_seconds
  > prometheus_rule_group_interval_seconds

Rule evaluation taking longer than the interval means alerts are delayed — a monitoring degradation that produces no obvious symptom.

Scaling

Growth	Response
More series	more Prometheus memory
Longer retention	more disk, or a remote write target
More queries	more CPU, recording rules
Rule evaluation lag	recording rules, longer intervals
Beyond one instance	Thanos, Mimir, or sharded scraping

# reduce series before adding hardware
ceph config set mgr mgr/prometheus/rbd_stats_pools ''

Quiz

Knowledge check · 4 questions

Q1. Why should both a time and a size retention limit be set on Prometheus?

Time-based retention is applied only when a block is compacted, so data stays on disk well past the configured window and the figure is only approximate A size limit keeps the retained set small enough to stay in the page cache, which is what keeps long-range dashboard queries fast A cardinality increase makes disk usage grow beyond expectation; the size limit discards old data rather than filling the disk and stopping ingestion Time limits are enforced by the exporter rather than by Prometheus, so metrics scraped from the Ceph manager are never covered by the window
Q2. A rule group taking longer to evaluate than its interval delays every alert in it while reporting no error.

True False
Q3. Plan monitoring capacity for a cluster expansion.

A 200-OSD cluster is tripling to 600 OSDs over the next year. The Prometheus instance has 8 GB of RAM and 100 GB of disk with 90-day time-based retention.
Q4. What is the first thing to try before adding Prometheus hardware for a series count problem?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set both time and size retention limits — the size limit is the backstop that prevents a cardinality change filling the disk and stopping ingestion. Alert on rule evaluation duration exceeding the group interval; it delays every alert and reports no error.

Cross-course references

Kubernetes: monitoring stack capacity must scale with cluster growth identically
Linux: any log or metric store needs a size bound as well as a time bound