HA Evidence

A team claims their observability stack is “highly available.” Two Loki ingesters run on different VMs. Both report up == 1. A “HA dashboard” shows green. Six months pass without incident. The first real failure takes down ingester A. The on-call engineer opens the runbook. The runbook says “HA is in place; no action required.” The on-call engineer waits. By 04:25, the team realises that the WAL on ingester A is gone, the ring has redistributed, and the recent writes that hashed to A are missing from the surviving replica. The HA was unverified. The dashboard was green because the dashboard was driven by the same processes that died. The first time the team needed the HA was the first time the team discovered they did not have it.

The discipline that prevents this is evidence: a test that proves the property, on a cadence that catches the regression before a real failure does.

What evidence looks like

Evidence is a record of an event that demonstrates the property. For HA, the event is “a single node failed and the system served correctly during and after the failure.” The record contains:

1. What was killed. Which node, which process, at what time.
2. What happened to the data plane. Did the surviving replicas continue to serve the queries the dead one served?
3. What happened to the write path. Did the distributor keep accepting writes? Did the WAL keep fsyncing on the survivors?
4. What happened to alerts. Did the alerting rule evaluator continue to fire (or resolve) during the failure window?
5. What happened to dashboards. Did the user’s panel continue to render at expected latency?

A test that does not include all five is not evidence. It is a checkbox.

The chaos test

The test is “kill one node of every stateful tier and observe the system.” The procedure is:

1. Snapshot the ring state of every stateful component.
2. Snapshot the WAL size and the chunk count of every ingester.
3. Pick one node of every tier at random.
4. Issue a coordinated kill (SIGTERM, then SIGKILL after 30s).
5. Observe the system for the next 5 minutes.
6. Capture: ring state, WAL size, chunk count, query latency,
   alert evaluations, /ready on every replica, /metrics on
   the LB.
7. Replace the killed nodes with fresh ones.
8. Observe the recovery: ring rebalances, WAL replays, fresh
   replicas join.
9. Compare to baseline.

Step 1 — snapshot the ring

# READ-ONLY
# Save the pre-test ring state. The diff after the kill tells
# you what was redistributed.
curl -s http://loki-distributor:3100/ring > \
  /tmp/ring-before-$(date +%s).json
curl -s http://mimir-distributor:8000/ring > \
  /tmp/mimir-ring-before-$(date +%s).json

Step 2 — snapshot the data plane

# READ-ONLY
# Baseline: how many streams does each ingester own? How much
# WAL is on disk? How many chunks in object storage?
for ing in ingester-1 ingester-2 ingester-3; do
  echo "$ing:" >> /tmp/dataplane-before.txt
  curl -s http://$ing.loki.svc:3100/metrics | \
    grep -E 'loki_ingester_streams|loki_ingester_wal_bytes' \
    >> /tmp/dataplane-before.txt
  ssh $ing "du -sb /loki/wal /loki/chunks" >> \
    /tmp/dataplane-before.txt
done

Step 3-4 — kill one node of each tier

# SERVICE-IMPACT / DESTRUCTIVE on the targeted pod
# Run inside an outage window approved by the change board.
kubectl cordon ingester-2 -n loki
kubectl drain ingester-2 -n loki --ignore-daemonsets \
  --delete-emptydir-data --grace-period=30
# After 30s the kubelet SIGKILLs any remaining containers.

Step 5-6 — observe

# READ-ONLY
# During the 5-minute observation window:
# - query latency
# - ring state diff
# - alert evaluations
# - /ready on every replica
# - replica count vs. expected
sleep 30 && \
  curl -s http://loki-distributor:3100/ring > \
    /tmp/ring-during.json
echo "ring diff:" && \
  diff /tmp/ring-before.json /tmp/ring-during.json | head -20

Step 7-9 — recover

# SERVICE-IMPACT
# Bring the killed nodes back. Observe the rebalance.
kubectl uncordon ingester-2 -n loki
kubectl rollout restart statefulset loki-ingester -n loki
sleep 300 && \
  curl -s http://loki-distributor:3100/ring > \
    /tmp/ring-after.json
echo "ring recovered:" && \
  diff /tmp/ring-before.json /tmp/ring-after.json | head -20

What a passing test looks like

A passing test produces this shape of evidence:

• Ring diff (during): the killed replica is removed; the remaining replicas absorb its partitions. Replication factor is preserved (3 active owners per partition).
• Query latency (during): within 2x baseline. The dip is acceptable; a 10x dip is a regression.
• Alert evaluation (during): every firing alert in the pre-test state remains firing (or resolves if it was already resolving). No alert moves to null because the rule evaluator is unreachable.
• /ready (during): every survivor returns 200. The killed node returns 503 until it is restarted.
• Ring diff (after): the ring is back to its pre-test shape with the same owner addresses.

A failing test produces a different shape. The most common failures are described below.

Failure modes the test catches

1. Replication factor drift. The killed replica was the sole owner of certain partitions because R was 1. The diff shows those partitions disappear from the ring. This is silent data loss in production.
2. Health-check false positive. The killed replica’s /ready continues to return 200 for 30 seconds because of stale probes. The LB routes queries to a dead replica and the user sees timeouts.
3. Compactor single point of failure. The compactor died at the same time as the killed ingester because they shared a node. The query path is fine; the compaction pipeline is now stalled.
4. Alertmanager split brain. Two Alertmanager replicas disagree on which alert should fire after the failure. The on-call engineer gets two pages for the same alert or zero pages for an alert that should be firing.
5. Recovery ring churn. The killed replica is replaced, joins the ring, immediately takes a partition, the WAL replay saturates the disk, and queries slow down for an hour after recovery. The test passes the “during” phase but fails the “after” phase.

The right cadence

The cadence depends on what you are testing and how often the system changes.

• Monthly: a full chaos test against the production stack during a low-traffic window. Captured in the runbook.
• Per change: any deployment that touches a stateful component’s config (replication factor, WAL settings, ring topology) triggers a scoped chaos test before the change is merged.
• Per quarter: a multi-node failure test (lose two ingesters at once). This is the boundary case the monthly test does not cover.
• Per infra event: any change to the underlying Kubernetes cluster, the Consul cluster, the S3 buckets, or the network fabric. The HA properties depend on all of these.

The cadence is not free. The monthly test takes ~30 minutes of an engineer’s time, including the write-up. The per-quarter multi-node test takes ~2 hours. The cost is the price of evidence.

How to record the evidence

A passing test is not evidence until it is written down. The record should live next to the runbook:

HA Test Record — YYYY-MM-DD
----------------------------
Tested: Loki microservices mode, Mimir microservices mode,
        Tempo microservices mode, Alertmanager cluster, Grafana
        replicas.
Nodes killed: ingester-2 (Loki), ingester-1 (Mimir),
              ingester-3 (Tempo).
Test window: 04:00-04:30 UTC.
Observers: on-call engineer + SRE manager.

Ring diff during: see attached.
Query latency during: 1.4x baseline (acceptable).
Alert evaluations during: 100% of pre-test firing alerts
                          remained firing.
Recovery time: 8m to ring steady state, 22m to WAL replay
               complete.

Pass/fail: PASS.

Next test: YYYY-MM-DD + 1 month.

The record is a regression detector. When the next month’s test shows query latency at 3.4x baseline, you can compare to last month’s record and know it is a regression, not a measurement of the normal degradation envelope.

What to do with the evidence

Three actions:

1. File the record. Put it where the runbook lives. The next on-call engineer reads the record and the runbook together.
2. Investigate the failures. Any failing test is a P2 even if the production system appears healthy. A failing HA test is a discovered vulnerability, not a known issue.
3. Update the SLO. The test’s measured recovery time is evidence for the alert-evaluation RTO. Use it.

Production guidance

Verification

You should now be able to answer:

• What is the difference between unverified HA and verified HA?
• What five things does a chaos test for HA need to measure?
• Why is “monthly” the right cadence, and why is “when we have time” the wrong one?
• What is the green-during-failure failure mode, and how do you avoid testing the wrong thing?

Quiz