High Availability Basics

A 02:14 alert fires: loki_ingester_replication_factor{zone="a"} 1. The single ingester in zone A is offline. Loki is rejecting writes that hash to that zone’s ring. Alerts stop firing from Grafana at 02:15 because the rules query the same ingester pool, which is also degraded. The on-call engineer opens a runbook written for “if Loki is down” and finds a section called High Availability that recommends running a second copy. A second Loki on a second host was indeed stood up six months earlier. The two instances are independent: each has its own ring, its own chunks, its own WAL. The “redundancy” is a parallel silo, not a replica. The user-facing behaviour is identical to running one instance: when one dies, half the data is unreachable and the alert storm reveals it.

The lesson is that high availability is a property of the system, not the deployment. Replicating a process is necessary but not sufficient. What you actually need is for the surviving instances to already hold the data that the dead instance owned, with no operator action required to recover it.

What “high availability” actually means

A production system is highly available for a given failure class when:

1. A single failure in that class does not interrupt the service.
2. The system returns to its steady state without manual data recovery.
3. The steady state preserves correctness: no data loss, no duplicate counts, no divergent views between replicas.

For the observability stack, the failure class is almost always “loss of one node.” Anything larger (loss of a whole rack, loss of a region) requires a separate conversation about failure domains and quorum. This lesson keeps the scope to single-node loss, because that is the failure that HA patterns are designed to absorb.

The phrase “high availability” without qualification is meaningless. A system is HA for something. Spell out the failure class, the recovery time, and the data-loss budget in every design document. “We need HA” without numbers is the starting point of an expensive misunderstanding.

Why a sysadmin cares

The two operational costs of not having HA are not symmetric:

• Loss of ingestion. Alerts stop firing because the alert rules cannot evaluate. The thing you were watching stops being watched. The incident you cared about becomes invisible.
• Loss of history. The dashboards load the last six hours instead of the last six weeks. The post-mortem cannot answer “when did this start?” because the answer is in a dead ingester’s WAL.

Either one is enough to fail an SLO. Both happen together when the failure is the process that owns the most recent data, which is the typical shape in a time-series database.

The third cost is the one nobody budgets for: the rebuild time. A dead stateful component has to be replaced, the ring has to rebalance, the WAL has to replay or the chunks have to re-ingest from object storage. That is hours of partial degradation even in the best-designed system.

Stateless vs stateful components

The single most important distinction in the observability stack is between components that hold no durable state and components that hold state that must outlive the process.

  Stateless replicas                Stateful replicas
  +-------------------+             +-------------------+
  | query             |             | ingester           |
  | distributor       |             | store-gateway      |
  | query-frontend    |             | compactor          |
  | querier           |             | index-gateway      |
  | ruler-querier     |             | alertmanager       |
  | gateway           |             | (with nflog/silto) |
  +-------------------+             +-------------------+
          |                                 |
          v                                 v
   "Add a replica,                  "Replication factor,
    point a load                     WAL, consistency,
    balancer at it."                 rebalancing."

Stateless components answer requests by reading state held elsewhere (object storage, peer ingesters, an external database). Scaling them is a matter of starting more processes and adding them to a load balancer.

Stateful components own part of the data. Scaling them is a matter of coordinating who owns what, replicating writes before they are acknowledged, and rebuilding ownership after a failure. The mechanisms are different, the failure modes are different, and the test procedure is different.

The four common shapes

Across the observability stack you will see four recurring shapes. Recognising which shape a given component uses tells you what HA costs and what it buys.

Shape 1: external database, stateless app

Grafana on MySQL/Postgres, Alertmanager without clustering, Tempo’s query-frontend. HA is database HA plus N replicas of the app behind a load balancer. The replicas hold sessions in memory only; everything that matters is in the database.

Shape 2: shared-nothing, hash-ring sharded

Loki’s distributors and ingesters, Mimir’s ingesters, Cortex ingesters, Tempo’s distributors. Writes are hashed to a ring of N owners, each write is replicated to R of them. HA is replication_factor and a write quorum. The cost is N times the per-replica storage and a small cross-node RPC.

Shape 3: shared store, cached query

Loki’s queriers and store-gateways, Mimir’s store-gateways, Tempo’s queriers. State lives in object storage (S3, GCS, Azure Blob, MinIO). Process-local caches accelerate reads. HA is “as many replicas as you want, all reading the same store.” A dead replica loses only its cache.

Shape 4: write-ahead log, periodic compaction

Prometheus 2.55’s local TSDB, Loki’s boltdb-shipper index, Mimir’s ingester WAL. State is local to the process. HA is a separate subsystem (Prometheus HA via Thanos or Mimir; Loki single-binary “simple mode” via replicated deploys; Loki microservices via ring replication).

What “the most common shape” is for each component

A useful starting point:

The table is a map of where the operational pain lives. The ingester row is the row that produces the most post-mortems.

How to validate you actually have HA

The validation that matters is not “are there two processes running?” It is “if I kill one, does the service answer and do the answers include the data the dead process owned?” Run the test, do not assume.

# READ-ONLY
# Confirm the replication factor and active owners of the
# stateful ring before the failure test.
curl -s http://loki-distributor:3100/ring | jq '
  .shards[] | {zone: .zone, owners: [.owners[].addr]}
' | head -40

The detailed test procedure lives in lesson 05. For this lesson, take the validation result seriously: the count of running processes and the count of logical replicas are not the same number.

Production guidance

The rest of Part LXVII builds the picture piece by piece. Lesson 02 covers the components that are trivial to scale. Lesson 03 covers the ones that require replication configuration. Lesson 04 names the silent failure shape that two instances without replication always produce. Lesson 05 defines the test that proves HA. Lesson 06 weighs the cost against the alternatives.

Verification

You should now be able to answer:

• Why is “we have two instances” not the same as “we have HA”?
• What is the operational difference between a stateless replica and a stateful replica in the observability stack?
• Which of the four HA shapes applies to a Loki ingester, a Tempo querier, a Grafana 11.x UI, and a Prometheus 2.55 single-binary?
• What is the first question to answer before scaling a component to N replicas?

Quiz