Stateful HA

A Loki ingester restarts after a routine OOM kill at 04:12. Its WAL files are intact on the local disk. The replica re-reads the WAL, reconstructs the in-memory series, and joins the ring again. The other two ingesters in the ring kept serving writes during the restart because each write was a triple replicated across the three owners. By 04:13 the restarted ingester is serving traffic again. The customer never noticed. This is what replication buys you. It also buys you a more subtle failure mode: a misconfigured replication_factor: 1 causes the same restart to lose the last 90 seconds of writes, with no log line loud enough to wake the on-call engineer.

Stateful HA is the lesson that prevents the second outcome.

Why “just add replicas” is the wrong answer

The stateful components in the observability stack — Loki ingesters, Mimir ingesters, Tempo ingesters, store-gateways, compactors — own a partition of the data. “Add a replica” is meaningless unless you also tell the system what the new replica should own, how many copies of each piece of data must exist, and how the system should recover when one of the copies dies.

Two mechanisms make this work:

1. A hash ring that maps each key (a stream ID in Loki, a tenant in Mimir, a trace ID in Tempo) to a set of owners. Each replica claims a contiguous range of the ring.
2. Replication of the write to multiple owners before the write is acknowledged to the client. The number of owners is the replication_factor.

Together they ensure that for every key, R replicas hold the data at any moment. When one replica dies, the system reshuffles the ring and brings a new replica up to R copies.

What the WAL is and what it protects

The write-ahead log is the file that ingester processes append every incoming write to before acknowledging the write. The sequence is:

client
   |
   v
distributor --hash(key)--> ingesters[R owners]
                              |
                              |  1. fsync to local WAL
                              |  2. update in-memory series
                              |  3. ack to distributor
                              v
                            disk

Three properties matter:

• Durability. Until the WAL is fsynced, the write is not durable. A power loss between the write arriving and the fsync loses the write. The fsync is the price of durability.
• Recovery. After a process restart, the WAL is replayed in order and the in-memory state is reconstructed. Without the WAL, the in-memory state has to be rebuilt from peers, which is slow and depends on every peer being reachable.
• Bounded loss. With WAL fsync on, the loss window is “what was buffered but not yet fsynced when the process died.” That is sub-second on a healthy disk. Without WAL fsync, the loss window is “everything since the last checkpoint.”

Configuring replication

The two settings that matter are replication_factor and the write quorum. The default for a single-node Loki install is replication_factor: 1 and the doc comment warns you to change it for production. Change it.

# loki.yaml — ingester section
ingester:
  lifecycler:
    ring:
      kvstore:
        store: consul
      replication_factor: 3        # copies per stream
      heartbeat_period: 5s
      heartbeat_timeout: 1m
    # JoinAfter: 0s makes the ingester accept traffic as soon
    # as it joins the ring. Set higher in dev so you can curl
    # /ready before the ring sees you.
    join_after: 0s
    observe_period: 0s
    final_sleep: 0s

  # WAL configuration
  wal:
    enabled: true                  # on by default in 3.x
    checkpoint_interval: 5m        # flush chunk state to disk
    flush_on_shutdown: true

The same shape applies to Mimir (ingester.ring.replication_factor) and Tempo (ingester.ring.replication_factor).

What “3” actually buys

With replication_factor: 3, every write goes to three ingesters. You can lose any two and still have a complete copy on the third. The cost is 3x the per-stream memory and 3x the per-stream WAL I/O. For most teams below ~5 TB/day, the cost is worth the protection. Above that, capacity planning changes (see lesson 06).

The write quorum is implied: with R=3 and the default quorum of quorum: false for Loki distributors, the write succeeds when at least one replica has fsynced. For a stronger guarantee, set the distributor’s replication factor to match and use distributor.ingestion_rate_limit and the distributor.write_failures counter to detect degradation.

How to validate replication is real

Three checks confirm the ring is configured correctly.

# READ-ONLY
# 1. Confirm the ring has the expected number of healthy owners.
curl -s http://loki-distributor:3100/ring | jq '
  .shards[] | {
    zone: .zone,
    state: .state,
    owners: [.owners[] | select(.state == "ACTIVE") | .addr]
  }
'
# expected: every shard lists three ACTIVE owners

# READ-ONLY
# 2. Confirm the replication factor is what you set.
curl -s http://loki-distributor:3100/config | \
  jq '.loki.config.ingester.lifecycler.ring.replication_factor'
# expected: 3

# READ-ONLY
# 3. Confirm every replica reports the same stream count for
#    a known stream after the write completes. Hash drift between
#    replicas indicates a misconfigured distributor hash.
for ing in ingester-1 ingester-2 ingester-3; do
  curl -s http://$ing.loki.svc:3100/metrics | \
    grep loki_ingester_streams
done
# expected: at least two of three replicas show non-zero count
# for any given stream; the third is the replicated copy.

Failure modes specific to stateful HA

The five modes that appear repeatedly:

1. Replication factor drift. An ingester is added with replication_factor: 1 left in the config (the default). The system silently treats the new replica as a singleton owner, and writes to its partition are not replicated. The failure surface: a restart of that replica loses its window.
2. WAL replay storm. A flaky disk produces WAL segments that fail to parse. Every replica that tries to recover from a shared WAL mount gets stuck in the same parse loop. Fix: per-replica WAL directories on local disk, never a shared NFS mount.
3. Ring rebalance storm. One zone disappears (a network partition). The ring redistributes the dead zone’s owners to the survivors. If the dead zone comes back, the redistribution happens again. With a WAL replay on top, the cluster spends more time rebalancing than serving. Fix: zone-aware replication with explicit failure-domain labels.
4. Divergent compactor. Two compactors run because the “one active, one standby” gate failed (KV store split brain). Each compactor rewrites the same index blocks. The result is duplicate work and an index that disagrees with itself. Fix: a real leader-election backend (etcd, Consul) and a fence token.
5. Clock skew on WAL timestamps. A replica with a clock 30s in the future writes WAL segments with future timestamps. The replay window for a later flush then miscalculates. Fix: run chrony on every node and alert on node_timex_offset_seconds.

How to troubleshoot a degraded stateful tier

The diagnostic order is:

1. Check the ring. GET /ring on the distributor or querier returns the owner set for every shard. The owner count per shard tells you whether the cluster has lost a replica.
2. Check the ingester health. GET /ready per ingester tells you which replicas are willing to accept traffic.
3. Check the WAL. du -sh /loki/wal per ingester tells you whether WAL is growing (good, normal) or has stalled (bad, flush is failing). The metric loki_ingester_wal_bytes_per_partition should be bounded.
4. Check the flush. loki_ingester_chunk_age_seconds tells you how old the unflushed data is. Above 2h you are reading from WAL, not from chunks.
5. Check the compactor. loki_compactor_running should be 1. Above 1 means split brain.

Security and performance implications

• Replication factor interacts with tenant isolation. A tenant whose streams are owned by ingesters that also own a noisy neighbour’s streams shares fate with that neighbour. Use ingester.partition_ring with explicit per-tenant partitions when isolation matters.
• WAL disk I/O is the bottleneck. Replication factor 3 triples WAL I/O. Plan SSD capacity at 2x peak WAL throughput to leave headroom for fsync storms.
• The compactor is a stateful singleton. It cannot be replicated active-active. The standby is hot only after the active dies; the active’s death costs the flush window.

Production guidance

Verification

You should now be able to answer:

• Why does the WAL exist and what does it protect?
• What does replication_factor: 3 actually buy in terms of failure tolerance?
• How do you confirm the ring has the expected number of active owners?
• What is the difference between replication and partitioning?

Quiz