The Cost of HA

The proposal reads: “Run a second Prometheus for HA.” The team approves the change. Six months later, the capacity bill is double what it was. The scrape load on every endpoint is double. The remote_write bandwidth is double. The query latency is double because the read path has to dedup across twice the data. The team did not budget for the doubling, and the on-call is now asking whether the second Prometheus was worth it.

HA is not free. The duplicate-scrape pattern from lesson 01 roughly doubles the cost on every layer. The trade-off is between availability (a single Prometheus is a single point of failure) and cost (the second Prometheus costs the same as the first). The discipline is to know the cost, to choose between duplicate-scrape, scrape sharding, and other HA patterns, and to make the choice deliberate.

What it is

The cost of HA in Prometheus is the resource cost of running a second instance that scrapes the same target set. The cost shows up on:

• CPU for scrape, ingestion, and the WAL append.
• Memory for the head block, the query engine, and the index.
• Disk for the local TSDB (the two-hour blocks) and the WAL.
• Network for scrape traffic (in and out of the Prometheus host) and remote_write bandwidth (out of the host to the long-term store).
• Target load for the endpoints that receive the doubled scrape traffic.

Every layer scales roughly linearly with the number of duplicate-scrape replicas. Two replicas cost twice as much as one. Three replicas cost three times as much. The total cost of HA is N * baseline_cost where N is the number of duplicate-scrape replicas.

The trade-off is between this cost and the availability gain. The availability gain is that a single Prometheus failure does not stop the alert path or the dashboard path. The cost is that every consumer of Prometheus data has to dedup or otherwise handle the duplication.

Why a sysadmin cares

Capacity planning for Prometheus is the difference between a predictable bill and a surprise. A team that runs a single Prometheus has a known cost: the resource cost of the host plus the storage cost of the long-term store. A team that runs two Prometheuses has roughly twice the cost on every layer. The “twice” is not exact (the second Prometheus has slightly lower cost because the dedup at the query layer reduces the effective data set) but is close enough that capacity planning should budget for the doubling.

Three questions drive the choice:

• What is the cost of a single Prometheus failure? If a single failure means the team is blind for an hour, the second Prometheus is worth the cost. If a single failure means the team reverts to a manual fallback for 15 minutes, the second Prometheus is a luxury.
• What is the cost of the second Prometheus? A 50 000 series Prometheus at 15s scrape interval costs roughly 4 CPU, 8 GiB memory, and 200 GiB disk per month. The second Prometheus costs the same. The doubled scrape load on the targets is harder to estimate.
• Is there a cheaper HA pattern? Scrape sharding (split the target set between replicas) gives HA with no duplication, at the cost of losing half the fleet when one replica dies. Remote_write-only (one Prometheus scrapes, the second Prometheus only receives remote_write) is cheaper but does not protect against scrape path failures.

How it works

The cost layers:

   +-------------------+         +-------------------+
   | Prometheus A      |         | Prometheus B      |
   |                   |         |                   |
   | scrape 15s        |         | scrape 15s        |
   | 50k series        |         | 50k series        |
   | ~4 CPU            |         | ~4 CPU            |
   | ~8 GiB memory     |         | ~8 GiB memory     |
   | ~200 GiB disk/mo  |         | ~200 GiB disk/mo  |
   +---------+---------+         +---------+---------+
             |                             |
       scrape (x2)                   scrape (x2)
             |                             |
             v                             v
   +---------------------------------------------------+
   |  Target Endpoints (node-exporter, app exporter)   |
   |  doubled request rate                              |
   +---------------------------------------------------+
             |                             |
       remote_write                  remote_write
             |                             |
             v                             v
   +---------------------------------------------------+
   |  Long-term Store (Thanos / Cortex / Mimir)         |
   |  doubled ingest rate                               |
   |  doubled storage for raw blocks                    |
   |  Compactor dedups blocks (saves on long-term)      |
   +---------------------------------------------------+

The CPU cost is dominated by the scrape and the WAL append. Each scrape allocates the new samples in memory, serialises them into the WAL, and updates the in-memory index. For a 50 000 series Prometheus at 15s scrape interval, the work is roughly 3 300 samples per second per replica. Two replicas double the work to 6 600 samples per second.

The memory cost is dominated by the head block. The head block holds the most recent two hours of samples in memory before they are written to a finalised block on disk. For a 50 000 series Prometheus with 15s scrape interval, the head block is roughly 4 GiB. Two replicas double the memory to 8 GiB on the second host (the memory is per-host, not shared).

The disk cost is dominated by the local TSDB. Each two-hour block is roughly 5-10 GiB for a 50 000 series Prometheus. Two hours * 12 blocks per day * 30 days = 360 blocks per month, or roughly 2-3 TiB per month per replica. Two replicas double the local disk cost.

The network cost is the scrape traffic and the remote_write bandwidth. For a 50 000 series Prometheus at 15s scrape interval, the scrape traffic is roughly 5 MB/s out (the Prometheus pulls from the target). Two replicas double the outbound traffic from the Prometheus hosts and double the inbound traffic to the targets. The remote_write bandwidth is similar.

Under the hood

How to configure it

There is no configuration to reduce the cost of duplicate scraping; the cost is inherent in running two instances that scrape the same targets. The configuration choices are about which HA pattern to use.

Pattern 1: duplicate-scrape. Two Prometheuses, identical config except replica label. Highest cost. Best availability.

# /etc/prometheus/prometheus-a.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: prod-eu
    replica: a
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.10:9100', '10.0.1.11:9100']

Pattern 2: scrape sharding. Two Prometheuses, disjoint target sets. Lower cost (each Prometheus scrapes half the targets), better availability (no duplicate scrape load on targets), at the cost of losing half the fleet when one Prometheus dies.

# /etc/prometheus/prometheus-a.yml (shard 0)
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.10:9100']

# /etc/prometheus/prometheus-b.yml (shard 1)
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.11:9100']

Pattern 3: remote_write-only. One Prometheus scrapes; the second Prometheus only receives remote_write from the first. Lower CPU and memory on the second Prometheus (no scrape load), but no protection against scrape path failures.

# /etc/prometheus/prometheus-b.yml
global:
  external_labels:
    cluster: prod-eu
    replica: b
# No scrape_configs. Prometheus B does not scrape.
remote_write:
  - url: http://prom-a:9090/api/v1/write
    # Read-only mirror of prom-a

Pattern 1 is the conventional “two replicas scraping the same targets” HA. Pattern 2 is the right pattern for cost- sensitive deployments that can tolerate a partial outage. Pattern 3 is the wrong pattern for HA (it does not protect against scrape failures) but is useful for read-replica deployments.

How to validate it

The first check is the Prometheus host’s resource usage:

# SEVERITY: READ-ONLY
top -bn1 -p $(pidof prometheus)

Expected output (illustrative):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 1234 prom     20   0  12.4g   8.1g  12.0m S  45.2  25.4   1234:56 prometheus

The %CPU should be well below the number of cores; the RES should be below the host’s memory budget. A Prometheus that is constantly above 80% CPU or above 80% memory is over its resource budget.

The second check is the scrape performance:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/metrics | grep -E 'prometheus_target_sync|prometheus_sd_kafka'
curl -s http://prom-a:9090/metrics | grep prometheus_target_scrape_pool_sync_total

Expected output (illustrative):

prometheus_target_scrape_pool_sync_total{scrape_job="node"} 432

A scrape pool that is falling behind shows a long interval between scrape attempts. The prometheus_target_scrape_pool_exceeded_label_limits_total metric rises when the scrape budget is exceeded.

The third check is the head block size:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/api/v1/status/tsdb | jq

Expected output (illustrative):

{
  "headStats": {
    "numSeries": 50234,
    "numLabelPairs": 6,
    "chunkCount": 1024,
    "minBlockTime": "...",
    "maxBlockTime": "..."
  }
}

The numSeries is the live series count in the head block. Compare to the expected count for the fleet; a divergence indicates a configuration change or a service-discovery drift.

The fourth check is the remote_write queue:

# SEVERITY: READ-ONLY
curl -s http://prom-a:9090/metrics | grep -E 'prometheus_remote_write|prometheus_wal'

Expected output (illustrative):

prometheus_remote_storage_queue_depth{url="..."} 2345
prometheus_wal_truncate_total 432

A growing prometheus_remote_storage_queue_depth means the remote_write cannot keep up; the WAL grows; eventually Prometheus drops samples on backpressure.

How it can fail

Six failure modes appear repeatedly when the cost of HA is underestimated.

1. The Prometheus host is undersized. The team runs two Prometheuses on a host sized for one. Symptom: the host thrashes on memory; the head block spills to disk; the WAL grows; scrape failures rise. The fix is to size the host for 2 * baseline, or to run each Prometheus on its own host.
2. Object storage cost is double. The long-term store receives twice the upload rate and stores twice the blocks before the Compactor dedups. Symptom: the storage bill roughly doubles for the first few hours of every two-hour block cycle, then the Compactor catches up. The fix is to budget for the peak, not the steady state.
3. The scrape load on a target is double its design budget. A node-exporter can handle 200 scrapes per minute; an application exporter may only handle 50. Symptom: scrape_duration_seconds on the second replica exceeds the scrape interval; up == 0 fires. The fix is to reduce the scrape interval, move one replica to scrape a disjoint set, or accept the cost.
4. Remote_write bandwidth is double. The remote_write target (Thanos Receive, Mimir, Cortex) cannot handle the doubled ingest rate. Symptom: Prometheus’s remote_write queue grows; the WAL grows; samples are dropped on backpressure. The fix is to scale the remote_write target.
5. The query layer slows down. The Thanos Querier has to dedup twice the series on every query. Symptom: dashboard panels load slowly. The fix is to scale the Querier (more CPU, more memory) or to use a smaller replica label set.
6. The Compactor falls behind. The Compactor has to dedup twice the blocks. Symptom: object storage accumulates overlapping blocks; the indexer slows down. The fix is to scale the Compactor or to accept the dedup happens at query time only.

How to troubleshoot it

The diagnostic order when the cost of HA is biting:

1. Inspect the Prometheus host’s CPU and memory. top -bn1 -p $(pidof prometheus) on each Prometheus host.
2. Inspect the head block size and series count. curl /api/v1/status/tsdb.
3. Inspect the remote_write queue and WAL size. The prometheus_remote_storage_queue_depth metric.
4. Inspect the long-term store’s ingest rate. The Thanos Receive or Mimir distributor metrics.
5. Inspect the scrape load on the targets. The exporter’s request count metric, if exposed.
6. Inspect the query latency on the read path. The Thanos Querier or Prometheus query metrics.

Security implications

The Prometheus host exposes the API and the scrape endpoint. A misconfigured reverse proxy without authentication exposes the API to anyone who can reach the port. The remote_write target requires authentication (basic auth, bearer token); a misconfigured remote_write URL that leaks the credentials is an information disclosure.

The scrape target authentication is per-exporter. A duplicate-scrape pair doubles the scrape traffic, but the authentication is per-request. The cost of authentication is paid twice.

Performance implications

The performance implications are the cost numbers themselves. A 50 000 series Prometheus at 15s scrape interval costs roughly:

• 4 CPU cores on the host.
• 8 GiB memory on the host.
• 200 GiB local disk per month.
• 5 MB/s outbound remote_write bandwidth.

Two duplicate-scrape replicas double each of these. The total cost is roughly 8 CPU, 16 GiB memory, 400 GiB disk per month, and 10 MB/s remote_write bandwidth. The scrape load on every target is doubled.

For larger fleets (10 million series, 5s scrape interval), the cost is proportionally higher. The trade-off is the same: HA doubles the cost; the team has to decide whether the availability gain is worth the cost.

Production guidance

• Size each Prometheus host for the baseline plus headroom. The rule of thumb is 2 bytes per sample per scrape interval for memory, and 1.3 bytes per sample per scrape interval for disk. For a 50 000 series Prometheus at 15s scrape interval over 15 days of retention: roughly 4 GiB memory and 100 GiB disk.
• Choose the HA pattern deliberately. Duplicate-scrape for maximum availability; scrape sharding for cost-sensitive deployments; remote_write-only for read-replica scenarios.
• Monitor the head block series count. Alert when it exceeds the expected count by 10%.
• Monitor the remote_write queue depth. Alert when it exceeds a configured threshold (10 000 by default).
• Monitor the host’s CPU and memory. Alert at 80% sustained.
• Budget for the storage peak. The Compactor dedups blocks but with a delay; the storage footprint peaks during the delay window.
• Use scrape sharding when the scrape load on a target is the limiting factor. Two Prometheus servers scraping disjoint halves of the fleet is cheaper than two scraping the whole fleet.

Verification

You should now be able to answer:

• What are the four cost layers that roughly double when a second duplicate-scrape Prometheus is added?
• Which Prometheus HA pattern gives the highest availability at the highest cost, and which pattern gives lower availability at lower cost?
• What is the rough resource budget for a 50 000 series Prometheus at 15s scrape interval (CPU, memory, disk)?
• Which Prometheus metric indicates the remote_write is falling behind?
• What is the role of the Thanos Compactor in reducing the long-term storage cost of duplicate-scrape pairs?

Quiz