Stateless Replicas

A Loki querier pod runs at 92% CPU. Its peers sit at 35%. The Service routes by iptables round-robin, so every request has equal probability of landing on the saturated pod. Within ten minutes the saturated pod starts 504-ing; the rate-limiter inside Loki drops queries that take longer than its timeout; users see “query timeout” on dashboards that worked ten minutes ago. The on-call engineer adds a second querier. The Service now distributes across two pods, the saturated pod still hits 92%, the second sits at 30%, and the same 504s continue. Two replicas is not the answer. Two replicas behind a load balancer that distributes unevenly is the same problem with extra capacity.

Stateless replicas are easy to scale in principle. They are also easy to scale wrong. This lesson covers the components that hold no durable state and the four things that decide whether your HA actually works.

What “stateless replica” means here

A replica is stateless for the purposes of this lesson when none of the following holds:

• The process owns a write-ahead log that must survive restart.
• The process owns a partition of the recent-data window that no other process owns.
• The process owns a long-lived cache that, if lost, would trigger a refill that exceeds the recovery time budget.

Anything that owns data lives in lesson 03. The remaining components are the read path, the dispatch path, and the stateless write-shard coordinator. Each can be scaled by adding a process behind a load balancer.

                        +---> query (stateless)
                        |
client --- LB ----------+---> querier (stateless)
                        |
                        +---> query-frontend (stateless)

                        +---> distributor (stateless)
ingester --- LB -------+
                        +---> distributor replica (stateless)

The arrows are load-balanced TCP or HTTP. The replicas do not talk to each other for correctness. They may talk to each other for caching or for gossip, but a dead peer does not lose data.

The list, by stack

The components you will replicate behind a load balancer:

Two rows in that list require care even though they are “stateless”:

• Grafana. Sessions live in the database, not in the process. As long as the database is reachable, N replicas behind a load balancer is a textbook stateless scale.
• Alertmanager. Silences and the notification log live on disk unless you configure a cluster peer list AND an external store. The default single-node alertmanager.yaml is not HA. The clustered configuration is, and is stateless with respect to the alerting pipeline itself.

How to configure stateless HA

The configuration has two halves: the component’s own listener and the load balancer in front of it.

Component side

Each replica reads the same config and exposes the same HTTP endpoints. Nothing about the config has to differ between replicas for the read path. The relevant checks:

# loki.yaml — querier section
querier:
  # MultiQuerier is on by default in microservices mode.
  # Each querier joins the ring of queriers and is discoverable
  # by query-frontends over the same ring.
  max_concurrent: 4   # concurrent in-flight queries per querier

common:
  ring:
    kvstore:
      store: consul    # memberlist also works for single-region
    heartbeat_period: 5s
    heartbeat_timeout: 30s

The ring is used for membership, not for state. Replicas that miss a heartbeat are marked unhealthy and removed from the LB pool; they do not lose data.

Load balancer side

For Loki queriers, a Kubernetes Service with cluster-local DNS is the simplest correct choice. Outside Kubernetes, HAProxy or Envoy with active health checks is the textbook option. The config below is HAProxy 2.8 with active HTTP checks against Loki’s /ready endpoint.

# haproxy.cfg (excerpt)
frontend loki_querier_front
  bind *:3100
  mode http
  default_backend loki_querier_pool

backend loki_querier_pool
  balance roundrobin            # round-robin is fine for Loki
  option httpchk GET /ready
  http-check expect status 200
  default-server inter 3s fall 3 rise 2
  server querier-1 10.0.0.11:3100 check
  server querier-2 10.0.0.12:3100 check
  server querier-3 10.0.0.13:3100 check

Three knobs matter:

• balance roundrobin is the right choice for stateless replicas with equal capacity. If replicas have different sizes, switch to balance leastconn or weight them.
• option httpchk GET /ready uses the component’s own health endpoint, not a TCP probe. Loki, Tempo, and Mimir all expose /ready that returns 503 during shutdown or while the process is still warming up.
• inter 3s fall 3 rise 2 means three consecutive failures to mark down, two consecutive successes to mark back up. This prevents flapping during a rolling deploy.

How many replicas?

The answer depends on three numbers and one policy.

1. Request budget per replica. A Loki querier on 4 cores can sustain roughly 4 concurrent split_queries_by_interval: 1d workloads before latency rises. A Tempo querier on 8 cores sustains more, but each trace query is heavier.
2. Peak query QPS. Measure with sum(rate(loki_request_duration_seconds_count[5m])) for Loki and the equivalent in Tempo.
3. Dependency fan-out. A query-frontend splits a 30-day range into 30 daily sub-queries, each of which fans out to queriers. The number of sub-queries per second is the actual load, not the number of user-facing queries.
4. The N+1 policy. Whatever number ceil(peak_qps / per_replica_budget) returns, add one. The “+1” exists so a single replica can be drained for upgrade without dropping requests.

A common starting shape that holds for most teams below ~5 TB/day:

These are starting numbers, not targets. Re-measure under load.

How to validate it

Three checks confirm the load-balanced stateless tier is healthy.

# READ-ONLY
# 1. Confirm the replicas all see the same config and version.
for h in querier-1 querier-2 querier-3; do
  curl -sf http://$h.loki.svc:3100/config | \
    jq -r --arg h "$h" '.loki.config.common.runtime + " on " + $h'
done
# expected: the same "loki" version string on every replica

# 2. Confirm the load balancer sees all replicas as healthy.
curl -s http://haproxy:8404/stats | grep loki_querier_pool
# expected: every server row shows status UP

The third check is the only one that proves the design works:

# READ-ONLY / SERVICE-IMPACT on querier-1
# Drain one replica and confirm queries still succeed.
curl -X POST http://querier-1.loki.svc:3100/ingester/ring?mode=LEAVE
# Then run a representative query from Grafana and confirm it
# returns within the latency budget.

Security implications

Stateless replicas terminate TLS, enforce auth, and apply per-tenant rate limits. A failure in any of these surfaces as a security incident even though the cause is HA-related.

• TLS termination. If you terminate TLS at the load balancer rather than at each replica, the LB becomes a single point of failure and a privileged credential holder. Run two LBs in active-active with VRRP or use anycast.
• Auth. Replicas that talk to an auth provider (e.g. Grafana asking JWT-bearers of an OAuth proxy) have a dependency that, if it fails, turns into “every query rejected.” Document the dependency and include it in HA tests.
• Per-tenant rate limits. A Mimir gateway holds the rate limit counters in memory; an HA design that drops gateway replicas on a rolling deploy resets the counters. If the counters matter (they usually do), use a shared store (Redis) for them.

Performance implications

• Cold caches. A fresh querier has no in-memory cache of recent series or trace IDs. The first N queries after a scale event hit object storage. Measure cache warm-up time separately from steady-state query latency.
• Connection storms. Kubernetes Services that route via iptables rebuild the conntrack entry on every pod restart. Use IPVS mode or an explicit load balancer for high-QPS paths.
• Asymmetric capacity. Round-robin on differently-sized replicas wastes the big replica’s CPU and saturates the small one. Weight or use leastconn.

Production guidance

Verification

You should now be able to answer:

• Which components in the observability stack are stateless, and which mechanism load-balances them?
• What is the difference between /healthz and /ready, and why does the load balancer probe matter?
• How do you decide the replica count for a stateless tier?
• What is the N+1 policy, and what failure does it prevent?

Quiz