When to Scale Metrics

A single Prometheus hits 80 percent CPU during the 4 pm scrape window; queries are timing out; the on-call is paging on up==0 for one in five scrape targets because the server cannot keep up with the scrape schedule. The team has spent the previous month debating whether to shard the workload or to introduce remote storage. By the time the alert fires, both options are late.

The right time to introduce remote storage — or to shard, or to federate, or to migrate to a different store — is before any of these symptoms. The threshold is not a single number; it is a set of measurable signals you watch over weeks, and a plan you execute before they fail.

What it is

When to scale metrics is the operational decision to move from a single Prometheus to a multi-host architecture. The decision has four shapes:

1. Vertical — bigger host. A single, larger Prometheus with more RAM, more cores, faster NVMe. Cheap; bounded by the single-writer invariant.
2. Functional sharding — multiple Prometheus, each scraping a subset of targets. Two Prometheus scraping the same targets for HA; or four Prometheus each owning a slice of the label space.
3. Remote write — Prometheus stays in place; samples are shipped to a long-term store (Thanos, Mimir, Cortex). The local TSDB shrinks to a recent-data role.
4. Federation — a top-level Prometheus aggregates rules from a fleet of lower-level Prometheus. Used when scrape fan-out is larger than a single host can hold; rare in 2026 because remote_write is usually cheaper.

The decision is which shape, in what order, against which signal. A team that gets the order wrong over-provisions hardware, runs into a cardinality cliff, or migrates to Mimir six months too early and pays the operational cost for a workload that did not need it.

Why a sysadmin cares

The cost of getting this wrong is paid in two currencies:

• Scaling too late. A crashed Prometheus, a lost head, a three-day gap in alerts. The on-call pays the price; the investigation reveals the thresholds were crossed weeks before.
• Scaling too early. A Mimir deployment that was needed at 50 M series but launched at 5 M series. The team now operates nine microservices, a Consul ring, and a Redis cache for a workload a beefier Prometheus could have held. The operational surface is paid whether the workload uses it or not.

The right answer is to watch the signals, define the threshold, plan the move, and execute it as a project — not as an emergency.

How it works

The thresholds, in four axes:

            Cardinality (active series)
 100 M -------------------------------------------
  50 M  ---  Mimir is the right answer
  10 M  ---  Thanos Sidecar + Store + Querier
   5 M  ---  single Prometheus on a beefy box
   1 M  ---  single Prometheus on a normal host
 100 K  ---  single Prometheus on a small host

            Sample rate (samples / sec sent into remote)
  1 M  ---  Mimir sized for the workload
 500 K  ---  Thanos or Mimir
 100 K  ---  single Prometheus is fine
  10 K  ---  single Prometheus is fine, well below capacity

            Query latency p99 (PromQL API, dashboards)
   5 s  ---  alert; investigate the cause
  30 s  ---  page; something has to give
  60 s  ---  data is effectively unavailable

The dominant variables are cardinality (cost of memory, index, storage) and sample rate (cost of CPU, network, ingest). Disk is usually a follower: it fills because cardinality grew, not because the workload intrinsically grew.

How to measure

The signals and the queries that surface them:

# Active series right now (the dominant cost variable)
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.headStats.numSeries'

# Samples per second being scraped
curl -s localhost:9090/metrics \
  | grep '^prometheus_tsdb_head_samples_appended_total'

# Active series (graph over the last 7 days; trend, not spot)
prometheus_tsdb_head_series

# Samples appended per second
rate(prometheus_tsdb_head_samples_appended_total[5m])

# Head block memory in bytes
prometheus_tsdb_head_series
  * on() group_left()
  3072                                  # ~3 KB per series in heap

# CPU saturating scrape work
rate(process_cpu_seconds_total{job="prometheus"}[5m])

# Query latency p99, last 5 minutes
histogram_quantile(0.99,
  sum(rate(prometheus_http_request_duration_seconds_bucket{
    handler="/api/v1/query"}[5m])) by (le))

# Disk used by the TSDB
node_filesystem_size_bytes{mountpoint="/var/lib/prometheus"}
  - node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"}

The capacity-planning exercise is to graph the first three over the last 90 days and project 6 months forward. The trend, not the spot value, is the input to the decision.

How to plan the move

The shape of the move depends on the workload.

Workload shape                Move
---------------------------  -----------------------------------
Cardinality climbing          sharding by label, or remote_write
                              to Thanos; same Prometheus process

Query latency climbing        first: reduce expensive queries,
                              add recording rules; then consider
                              Querier replicas (Thanos) or
                              Query-frontend (Mimir)

Single-host durability        HA replica + remote_write to a store;
                              do not wait for the host to die

Multi-tenant SaaS             Mimir from day one; Cortex is the
                              wrong answer at this scale

A practical migration sequence for a team moving from a single Prometheus to remote_write to Mimir:

Week 1     Instrument. Add the capacity signals to a dashboard.
           Document the baseline.

Week 2-3   Stand up the remote store in dev. Remote_write the
           dev Prometheus at low volume. Validate samples reach
           the store.

Week 4-6   Add a second Prometheus as an HA replica. Verify
           distinct external_labels. Remote_write both to the
           dev store. Validate dedup.

Week 7-9   Promote the store to staging. Remote_write the
           staging Prometheus at full volume. Validate query
           latency against last week's data.

Week 10-12 Promote to production. Remote_write behind a flag
           (initially disabled) to allow rollback. Switch
           dashboards to query the store. Decommission the
           local-only data path only after the dashboards
           agree.

The discipline is: every move is reversible for a window of weeks; the cutover is a feature flag, not a config change.

How to validate it

# The capacity signals are being collected and graphed
curl -s grafana/api/dashboards/uid/capacity \
  | jq '.dashboard.panels | length'

# The remote_write pipeline is healthy
curl -s localhost:9090/metrics \
  | grep -E 'prometheus_remote_write_samples_(sent|pending|dropped)_total'

# The remote store sees the same series as the local Prometheus
curl -s mimir-frontend:8080/api/v1/label/__name__/values \
  -H 'X-Scope-OrgID: tenant-42' \
  | jq '.data | length'
# Compare to the local Prometheus
curl -s localhost:9090/api/v1/label/__name__/values \
  | jq '.data | length'

Five metrics worth alerting on before the move:

# Active series crossing 60 percent of host RAM budget
prometheus_tsdb_head_series > 0.6 * 5_000_000

# Scrape-target failures climbing
rate(prometheus_target_scrapes_exceeded_body_size_limit_total[5m]) > 0

# Query p99 above the alert threshold (5 s typical)
histogram_quantile(0.99,
  sum(rate(prometheus_http_request_duration_seconds_bucket{
    handler="/api/v1/query"}[5m])) by (le)) > 5

# WAL replay time climbing on restart
increase(prometheus_tsdb_wal_replay_duration_seconds[1h]) > 60

# Remote_write samples_pending climbing
prometheus_remote_write_samples_pending
  > 0.5 * (prometheus_remote_write_queue_capacity * max_shards)

How it can fail

1. Capacity signals never instrumented. Symptom: the team discovers the threshold was crossed when the Prometheus crashes. There is no trend line, no plan, no early warning. The post-mortem says “we should have seen this coming”, and the next team inherits the same gap.
2. Scaling too early to Mimir. Symptom: a Mimir cluster with Consul, Redis, and nine microservices for a workload that a single Prometheus on a 64 GB host could have held. The operational cost is paid whether or not the workload uses the capacity. The fix is to size to the actual workload, not to the imagined one.
3. Scaling too late with a single host. Symptom: a head block at 32 GB RAM; CPU saturation at the scrape window; up==0 alerts firing on the Prometheus’ own targets because the server cannot keep up. The fix is already an incident.
4. HA replica labelling mistake during scale-out. Symptom: two Prometheus writing the same remote_write target with the same external_labels. The remote store receives duplicate samples; query-time dedup hides the duplication; storage bills double. Same discipline as the remote_write lesson; this is the most common scale-out mistake.
5. Recording-rule growth making the rule evaluator the bottleneck. Symptom: the scrape work fits; the rule evaluator does not. CPU is at 100 percent during rule windows; queries are fast but new rules cannot be added. The fix is to move rules out of the local Prometheus (Mimir Ruler, Thanos Ruler) or to scope them down.
6. Migration without a rollback window. Symptom: the cutover is irreversible; an undetected regression is only noticed days later; the team has no way to compare the old platform to the new. The fix is the feature flag + dual- write window; the rollback is the discipline, not a button.

How to troubleshoot it

1. Are the capacity signals being collected? If not, the decision is being made blind. Instrument first; the trend line is the input.
2. What is the dominant axis? Cardinality, sample rate, query latency, or rule evaluation cost? The fix is shaped by the axis.
3. Is the move reversible? If the answer is “no”, the move is a deployment, not a project. Plan a rollback before the cutover.
4. Is the new platform doing what was promised? Compare query latency, sample acceptance rate, and dashboard correctness between the old and new platforms during the dual-write window.
5. Is the team trained? A Mimir cluster is a different operational surface than a single Prometheus. The migration plan includes training, runbooks, and on-call handover.

Security implications

• Multi-host = wider blast radius. A misconfigured remote_write that leaks one tenant’s ID writes another tenant’s data. A single Prometheus is bounded by its host; a fleet is bounded by the team’s operational discipline. Tenant isolation becomes a security control, not just a billing control.
• Network reachability of the remote store. A remote_write endpoint on the public internet is a write- only credential to inject arbitrary metrics into the platform. The store must sit on a private network, behind an authenticating proxy, or both. A misconfiguration that binds to 0.0.0.0 on a public IP is a credential leak.
• Hash-ring KV store (Mimir / Cortex). Consul or etcd holds the cluster topology. Network access to the KV store is read access to the membership of every stateful component. The KV store should sit on a private network, with mTLS and authn.
• Bucket credentials. The store needs S3 (or equivalent) credentials. Prefer IAM roles; restrict DeleteObject to the Compactor; rotate access keys. A leaked Sidecar credential can read every block in the bucket.

Performance implications

The dominant scaling decisions are:

Cardinality        RAM (head) and index size; the first to hurt
Sample rate        CPU (scrape + send); second to hurt
Query latency      Dashboard CPU; usually last to hurt, first
                   to be noticed
Rule evaluation    CPU; can saturate before cardinality does

The trade-offs of each move:

Vertical                 cheaper; bounded by host; transient relief
Functional sharding      moderate cost; horizontal; loses single-
                         host query surface
remote_write to Thanos   moderate cost; object storage economics;
                         does not help query latency until you add
                         Querier replicas
remote_write to Mimir    higher cost; multi-tenant native;
                         horizontal query and ingest
Federation               lowest cost when needed; rarely needed in
                         2026 because remote_write is usually
                         cheaper

The right move at the right time is the cheapest move that sustains the workload for the planning horizon. Six months of project runway is the practical target.

Verification

You should now be able to answer:

• What are the four measurable signals that indicate a single Prometheus is at its limit?
• At what cardinality does a single Prometheus stop being viable, and at what cardinality does Mimir become the right answer?
• What is the difference between scaling too early and scaling too late, and how does each fail?
• Why must the cutover to a remote store be reversible for a window of weeks?
• What is the first metric to instrument if a team has instrumented nothing?

Quiz