HA Cost Trade-offs

A platform team provisions “full HA” for the observability stack. Three ingesters per zone, three zones, R=3 replication across zones. Six months later the observability bill is 38% of the total platform spend. The CFO asks what the spend buys. The team realises that two of the three zones’ worth of replicas were added before any SLO demanded them, that one of the three tenants whose data drives the spend could tolerate a 30-minute restore from object storage instead of zero-loss replication, and that the dev/test tier of the stack is using the same R=3 ring as production. HA is not free, and “HA everywhere” is rarely the correct answer.

The right model is per-tier HA policy, matched to the outage cost of the data.

What HA actually costs

The cost of the second replica is not “double the first.” It is:

• 2x the per-replica compute for the same data. CPU, memory, network, and storage are all doubled.
• 2x the WAL I/O for stateful ingesters. The fsync cost scales with the replication factor.
• Shared object storage cost. Object storage grows with R, not with logical data. With R=3, you pay for three copies in object storage.
• Cross-zone network. In a multi-zone deployment, every replicated write crosses two zone boundaries. Egress between zones is the most expensive line item on the bill for many clouds.
• Operational complexity. More replicas means more upgrades, more chaos tests, more on-call pages, more careful capacity planning.

A worked example for Loki at ~2 TB/day ingest:

Loki ingester R=3 single zone (3 replicas):
  CPU     : 3 x 4 cores       = 12 cores
  Memory  : 3 x 16 GiB        = 48 GiB
  WAL SSD : 3 x 500 GB NVMe   = 1.5 TB
  Object  : ~6 TB (R=3 of 2 TB logical)

Loki ingester R=3 multi-zone (9 replicas):
  CPU     : 9 x 4 cores       = 36 cores
  Memory  : 9 x 16 GiB        = 144 GiB
  WAL SSD : 9 x 500 GB NVMe   = 4.5 TB
  Object  : ~6 TB (same logical data, but cross-zone egress
                    per write)
  Network : 2 zone crossings per write, ~80 GB/day egress

The multi-zone layout costs roughly 3x the single-zone R=3
layout, mostly in compute and cross-zone network.

The same shape applies to Mimir, Tempo, and Prometheus-via- Thanos. The numbers move; the proportions do not.

The trade-off framework

For every tier of data, ask three questions:

1. What is the outage cost? If the data is unavailable for 30 minutes, what does it cost the business? In dollars, in pages, in compliance exposure?
2. What is the loss cost? If 30 minutes of data is permanently lost, what does it cost?
3. What is the HA cost? What does it cost in compute, storage, and network to reduce the outage cost to zero and the loss cost to sub-second?

The answer drives the policy:

The “tested restore” column matters. A single-instance deployment without a tested restore is a single point of failure. A single-instance deployment with a tested restore is a deliberate cost optimisation, not a mistake.

Per-tier guidance

Hot tier: production observability for production services

The hot tier is the data that drives on-call decisions. Loss or outage costs real money in incident response time. The correct policy is:

• Loki: microservices mode, R=3 within zone, R=3 across zones if the business can afford it, Consul/etcd KV.
• Mimir / Cortex: microservices mode, R=3 by default, zone-aware ingester ring.
• Tempo: microservices mode with S3 backend, R=3 ingester, distributor fronting.
• Prometheus: Thanos sidecar with R=3 receive on ThanosReceiver, or Mimir remote_write. The single-binary “HA” pattern is the wrong answer here.
• Grafana: 2 replicas behind LB, shared DB.
• Alertmanager: 3 peers in gossip cluster, nflog DB external.

Warm tier: staging and pre-production

The warm tier is data you would want if staging had a production-class incident. The outage cost is medium; the loss cost is low (the production environment is the source of truth). The correct policy is:

• Loki: microservices mode, R=2 within zone. Accept that one node can be lost without data loss, but two nodes cannot.
• Mimir / Cortex: microservices mode, R=2.
• Tempo: microservices mode, R=2.
• Prometheus: Thanos sidecar with R=2, or single-binary with verified backup-and-restore.
• Grafana: single replica is acceptable if staging is allowed to be offline during incidents.

Cold tier: development, demos, and ephemeral environments

The cold tier exists to make engineers’ lives easier. The outage cost is low; the loss cost is near zero. The correct policy is:

• Loki: single-binary mode with a daily object-store snapshot. No ring, no replication.
• Mimir: not appropriate at this scale. Use a Prometheus single-binary with a daily snapshot.
• Tempo: single-binary with backend: local is fine for short-lived demo traces. Object-store backend if traces need to survive a node failure.
• Grafana: single replica, SQLite or embedded DB.
• Alertmanager: single replica. Accept that alerts may be lost during the demo.

The cold tier is the place where most teams over-spend on HA. A dev environment that mirrors production’s R=3 multi-zone ingester ring is paying three zones of compute to protect data that nobody would notice losing.

How to right-size an existing deployment

The cost review is a checkpoint. The trigger is one of:

• The observability bill crossed a budget threshold.
• A new tenant is onboarded with a different SLA.
• A cluster is decommissioned (downgrade opportunity).
• A migration (e.g. simple-mode to microservices) is planned.

The review procedure:

1. Inventory the tiers. List every cluster, every tenant, every environment. Tag each as hot, warm, or cold.
2. Match policy to tier. Apply the table above. Anything running R=3 multi-zone in the cold tier is over-provisioned.
3. Downgrade the lowest-priority tier first. The cold tier’s replicas are the cheapest to remove.
4. Verify with the chaos test. After the downgrade, run the chaos test for the affected tier. The test still has to pass within the tier’s documented SLO.
5. Document the decision. Write down the tier, the policy, and the cost. The next review starts from this record.

What HA does not buy

A second replica does not buy:

• Performance. Replication is not parallelism. The query throughput of an ingester ring with R=3 is the same as R=2 for the same query load. If you need more query throughput, scale the queriers, not the ingester ring.
• Durability against bugs. If the ingester corrupts its WAL because of a software bug, R=3 means three copies of the corruption. Replication protects against the failure of hardware, not the failure of code.
• Protection against operator error. Replication does not undo a DELETE from object storage. Object-store-level backup (with versioning and a tested restore) is what protects against operator error.
• Lower query latency. Adding replicas to the ingester ring does not make queries faster. The querier is the read path. Scale the queriers if latency is the problem.

Production guidance

Verification

You should now be able to answer:

• What is the cost of the second and third replica of a stateful ingester?
• What three questions determine the right HA policy for a tier?
• Why is “HA everywhere” the wrong default?
• What does HA not buy?

Quiz