Break/Fix: Observability Stack Loses a Node · Break/Fix · Observability

Reported symptoms

At 03:40 the hypervisor hosting obs-02 took a hardware fault and did not come back. obs-02 is one of three nodes running the observability stack: three Loki ingesters, three Tempo ingesters, three Grafana replicas behind a load balancer, three Alertmanagers, two Prometheus servers. A replacement VM was built and joined the ring at 04:26.

By 04:30 the platform looked entirely recovered, and the incident was closed at 05:10 as “node failure, HA worked”. Four things came in afterwards, over three days, from three different people.

A thirty-second Grafana outage. At 03:40 Grafana returned 502 for about half a minute even though two of its three replicas were healthy the whole time. Filed as “Grafana HA does not work”.

Two pages for one alert. The on-call engineer was paged twice for the same alert, four minutes apart, with the same summary. Filed as “Alertmanager duplicate notifications”.

An hour of slow dashboards after everything was fine. Query latency sat three to four times baseline from 04:26 to about 05:20 — beginning when the replacement node joined, not when the old one died. Filed as “queries slow after maintenance”.

Something does not add up in the incident review. Three days later, someone writing up an unrelated application incident from that night finds the log volume for 02:00 to 03:30 looks thin. The collectors say they shipped a certain number of log records; a count_over_time over the same window returns about two thirds of it.

And the thing that is not a symptom: Prometheus is flawless for the whole window, including the metrics it scraped from obs-02 right up to 03:40. Half the platform has a perfect record of a night the other half has partially lost. Nobody was paged about missing data at any point, and ingestion recovered by itself about a minute after the node was lost.

Evidence provided

Start with the accounting, because it is the only symptom that implies permanent damage and the only one that cannot be argued away. Ask the collectors what they sent, and ask whether any of it failed.

Read-only / Safeevery record was accepted; nothing failed to send

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum(increase(otelcol_exporter_sent_log_records[2h]))' \
--data-urlencode 'time=2026-08-11T03:40:00Z' \
| jq -r '.data.result[0].value[1]'
curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum(increase(otelcol_exporter_send_failed_log_records[2h]))' \
--data-urlencode 'time=2026-08-11T03:40:00Z' \
| jq -r '.data.result[0].value[1]'

41295310
0

Illustrative output

Then ask the platform to return the same window.

Read-only / Safeabout a third of what was accepted is not there

$ curl -sG http://loki-gateway.internal.example.com:3100/loki/api/v1/query \
-H 'X-Scope-OrgID: prod' \
--data-urlencode 'query=sum(count_over_time({cluster="prod"}[2h]))' \
--data-urlencode 'time=2026-08-11T03:40:00Z' \
| jq -r '.data.result[0].value[1]'

27498004

Illustrative output

Traces show the same proportion, and the shape of the loss is the tell: the traces that survive are complete. Nothing is partially missing.

Read-only / Safewhole traces present or whole traces absent, never partial

$ for id in $(cat /tmp/trace-ids-from-window.txt); do
code=$(curl -s -o /dev/null -w '%{http_code}' \
  "http://tempo.internal.example.com:3200/api/traces/$id")
echo "$id $code"
done | awk '{c[$2]++} END {for (k in c) print k, c[k]}'

200 134
404 66

Illustrative output

Now ask the ring who owns what. This is one command, it needs no incident context, and it answers the question the HA dashboard was never asked.

Read-only / Safeone owner per shard, and this is the healthy steady state

$ curl -s http://loki-distributor.internal.example.com:3100/ring \
| jq '[.shards[] | {zone: .zone, owners: [.owners[] | select(.state == "ACTIVE") | .addr] | length}] | .[0:4]'

[
{ "zone": "a", "owners": 1 },
{ "zone": "a", "owners": 1 },
{ "zone": "a", "owners": 1 },
{ "zone": "a", "owners": 1 }
]

Illustrative output

Read-only / Safethe value the single-binary install ships with

$ curl -s http://loki-distributor.internal.example.com:3100/config \
| jq '.loki.config.ingester.lifecycler.ring.replication_factor'

Illustrative output

Two supporting facts. loki_ingester_streams on the two surviving ingesters did not rise by a third after 03:40, so the lost streams were not picked up from anywhere — they were simply gone. alertmanager_cluster_health_score dipped at 03:40 and recovered without anyone touching it. The ingester config has not been changed since the original single-node install, and there is no record of a failure test ever having been run against this stack.

Work the evidence before reading on

Three loud symptoms, one quiet one, and a stack that has three of everything.

The collectors report zero send failures and the distributor confirms it accepted the data. Where can data go after it has been accepted and acknowledged but before it reaches the object store, and how many copies of it are there while it is in that place?
The missing window is the two hours before the node died, not the forty-six minutes it was gone. What has a two-hour boundary in this stack, and why does that boundary decide what was lost?
Traces are missing whole or present whole, never partial. What does that tell you about how work is distributed, and would you expect the same shape if the cause were a network problem or a disk problem?
Three of the four symptoms cleared themselves without anyone doing anything. Is self-healing evidence that those three were the same fault as the fourth?

Before continuing: the ring reported one owner per shard before the incident as well as during it. Given that, what would this stack have to lose for the loss to be zero, and what does the phrase “three-node HA cluster” actually promise here?

Root cause

1. Three instances, one copy

replication_factor was 1.

That is the value a single-binary Loki install ships with, and it is correct for a single-binary install. When this deployment grew from one node to three, the ingester config was not revisited. It has not been changed since.

With a replication factor of one, the hash ring maps every stream and every trace to exactly one owner. Three ingesters divide the keyspace into three parts and each part has a single holder. The ring is doing precisely what it was told: it is sharding, and it is not replicating. Adding processes made the platform bigger. It did not make any piece of data exist twice.

This is why the trace losses are whole traces rather than partial ones. Tempo shards by trace ID, so every span of a given trace lands on the same owner. Lose the owner and you lose the trace entirely; the traces that survive are untouched. A network fault or a disk fault would have produced ragged, partial damage. A clean partition of the keyspace produces exactly what the evidence shows.

2. The lost window is two hours because that is the WAL’s job

The missing data is not the traffic during the outage. It is the traffic from the two hours before it.

An ingester acknowledges a write once it is fsynced to the local write-ahead log, then holds it in memory and flushes it to object storage periodically. Between the acknowledgement and the flush, the data exists on that ingester and nowhere else — that is the gap the WAL exists to bridge, and it is about two hours wide.

A process restart is survivable: the WAL is on disk, it replays, the state comes back. That is what the WAL protects against, and it is why teams with replication_factor: 1 can restart ingesters for years without noticing anything wrong.

obs-02 did not restart. The host was lost and did not return, so its disk went with it. Every acknowledged-but-unflushed write that hashed to obs-02 was destroyed, along with the only copy of each one. The collectors had long since released those records, because the platform told them the write had succeeded.

Traffic during the outage is fine, which is the detail that made this look survivable at 04:30. Once the ring marked obs-02 unhealthy, the distributor re-hashed its keys onto the survivors and ingestion resumed on its own. That is real recovery, and it recovered the wrong thing: the future, not the past.

3. The three loud symptoms are the system working

None of the other three tickets is this fault, and it is worth saying so precisely rather than leaving them attached to it.

The Grafana blip. Grafana with an external database is a genuinely stateless replica set; state lives in the database, not in the process. The load balancer needs a few consecutive failed health checks before evicting a replica, so for a short window a fraction of requests still route at a node that has just died. Thirty seconds of 502 is that window. Shortening it is a tuning decision with a flapping cost on the other side, not a defect.

The duplicate page. Alertmanager replicas gossip their notification log so a given alert notifies once. An instance that sends a notification and dies before that entry propagates leaves a peer with no record of it, and the peer sends again. Two pages for one alert is the cost of not missing the page entirely, and the alternative failure — zero pages — is far worse.

The hour of slow queries after recovery. This one starts when the replacement joined at 04:26, not when the original died. Joining a ring triggers a redistribution, and a fresh replica replays its WAL while taking on new ownership; both compete for the same disk. That hour is a cost to plan for and test, not a fault.

Under the hood

The reason nobody was paged for the only thing that mattered is worth pulling apart, because the same gap exists in most stacks that believe they are highly available.

The HA dashboard counted processes. Three ingester processes were running; the panel read 3/3. When one died it read 2/3, and when the replacement joined it read 3/3 again. Every one of those readings was accurate, and none of them was about data. A process count cannot distinguish three replicas of one logical service from three independent shards of it, and that distinction is the entire difference between a node loss and a data loss.

The ring can distinguish them, and it will tell you for free. The distributor’s ring endpoint reports the owners of every shard, and the number that matters is how many are ACTIVE. Three ACTIVE owners per shard means you can lose one and keep serving. One ACTIVE owner per shard means every shard is a single point of failure, and it means that on an ordinary Tuesday with nothing wrong — which is exactly when this panel would have been amber for the eighteen months before the fault.

There is a second gap behind the first. Nothing here ever reconciled what was accepted against what could be read back. Every error rate was zero because there were no errors: the writes succeeded, and then the data ceased to exist. A failure that produces no error is invisible to every signal except a comparison of two independent counts, and nobody was making that comparison until an unrelated review noticed the log volume looked thin three days later.

Raising the replication factor is not a free change

The fix is one config value, which makes it look like a five-minute job. It is not.

Changing replication_factor requires restarting the ingesters, and while that rollout is in flight the replication factor is still 1. Every ingester you restart is, for the duration of its restart, the sole owner of a third of the recent window. You cannot safely roll the tier until you can survive losing one of them, and you cannot survive losing one of them until you have rolled the tier.

The way through is to make each restart graceful rather than abrupt. An ingester configured to flush on shutdown, and given a termination grace period long enough to finish, pushes its unflushed window to object storage before it exits — so there is nothing left in it to lose. Take one ingester at a time, and wait for the ring to show it ACTIVE again and its WAL replayed before starting the next. A rolling restart that runs two at once against a replication factor of 1 is how a remediation becomes a second incident.

The other cost is capacity. Three copies means roughly three times the per-stream memory and three times the WAL I/O. If that headroom is not there, the rollout will OOM partway through and leave you with a partially converted tier, which is worse than either end state. Measure first.

Resolution

Establish and publish the loss before fixing anything. Reconcile what the collectors sent against what the platform returns, per signal, for the affected window. The data is not recoverable — the collectors acknowledged it and released it, and the object store never received it — so the only useful output is an accurate statement of what is missing, given to anyone reviewing that night. A query that silently returns two thirds of its input is worse than one that errors.
Detach the three benign tickets. The Grafana blip, the duplicate page and the post-recovery latency are correct behaviour under a node loss. Leaving them attached to this incident invites three changes that each make something else worse.
Measure the headroom before touching the ring. Three copies means roughly three times the per-stream memory and WAL I/O on every ingester. If it is not there, stop here and hold.
Set replication_factor: 3 on the Loki and Tempo ingester rings, and confirm each ingester is configured to flush on shutdown with a termination grace period long enough to complete the flush.
Roll the tier one ingester at a time, gracefully. Wait for the ring to report the restarted ingester ACTIVE and its WAL replayed before starting the next. Never restart two while the replication factor is still 1.
Confirm the ring reports three ACTIVE owners per shard afterwards. Configuration and ring state can disagree, and the ring is the one that decides what happens when a node dies.
Replace the process-counting HA panel with one that reads the owner count per shard from the ring. The old panel was accurate and useless; make the new one answer the question that was actually being asked of it.
Schedule the failure test. Until a node has been killed deliberately and the result recorded, the change above is a belief rather than a property.

Verification

The ring reports three ACTIVE owners for every shard, and the running config reports replication_factor: 3 on every Loki and Tempo ingester. Check both: they can disagree, and the ring is authoritative.
Kill a node, not a process. Killing a process tests the application; killing a node tests the orchestrator, the load balancer, the ring, the WAL and the humans, which is where the failures that matter actually live.
During the failure, writes continue to be accepted and a query over the recent window still returns the streams the dead node owned. That second half is the whole test — the first half was already true when the replication factor was 1.
During the failure, rule evaluation continues and alerts that were firing stay firing. An alert that goes to no-data because its evaluator is unreachable is the failure mode that hides the next incident.
After the failure, reconcile accepted against queryable for the test window. The number that matters is that the two match, not that the dashboards went green again.
Watch the recovery phase as closely as the failure phase. A churning ring and a WAL replay that saturates the disk will pass the during phase and fail the after phase.
The new ring-based HA panel goes amber during the same test and returns to green afterwards. A panel that stays green throughout a deliberate node kill is the panel that failed this incident.

Prevention

Set the replication factor at provisioning time and treat the single-node default as a trap rather than a starting point. It is correct for the install it ships with and wrong for every deployment that grows past it, and the growth is exactly the moment nobody revisits the config.
Alert on owners per shard, read from the ring. A process count cannot tell three replicas from three shards, and that distinction is the whole difference between an outage and a permanent loss. The ring endpoint answers it in one request and would have been amber every day for eighteen months.
Reconcile accepted against queryable on a schedule. This class of failure produces no errors at all — the write succeeded and then the data stopped existing — so it is invisible to every error rate and every availability check. loki-canary writes a synthetic line and reads it back, and its round-trip metrics catch the shape; be honest that it only covers the streams the canary itself owns, so it complements the ring check rather than replacing it.
Test the failure on a cadence. High availability is a claim about behaviour under a condition that has not happened yet. This stack had never been tested, and the first test was conducted by a hypervisor at 03:40 with the on-call engineer as the only observer.
Express the failure domain. Zone-aware replication with explicit failure-domain labels stops three replicas landing behind one hypervisor — a distinct problem that a correct replication factor does not solve.
Separate the loud symptoms from the damaging ones during the review. Three tickets self-healed and one destroyed data, and the three that self-healed are the ones that got filed. A review that ranks by how much noise a symptom made will fix the wrong three things.

What if obs-02 had simply rebooted and come back with its disk?

Nothing would have been lost, and nobody would have learned anything.

The WAL would have replayed on startup, the in-memory state would have been reconstructed, the ingester would have rejoined the ring and flushed its window to object storage as normal. The incident would have been the thirty-second Grafana blip, the duplicate page and an hour of slow queries — three benign symptoms, one closed ticket, and a stack still one hypervisor fault away from losing two hours of data.

That is how a replication factor of 1 survives for years. It tolerates every failure that returns the disk — process crashes, OOM kills, graceful reboots, kernel upgrades — and none of the failures that take the disk away. You do not get to choose which kind you get.

The uncomfortable corollary is that years of uneventful restarts were not evidence of anything. They were a sample drawn entirely from the failure class this configuration happens to tolerate, and the platform’s confidence had been built on it.

Why this matters

The most expensive thing in this incident is not the lost data. It is that the platform reported success throughout, and every individual report was accurate.

The collectors were told their writes succeeded, and they were. The distributor accepted the data, and it did. Ingestion recovered by itself, and it had. The HA dashboard counted three processes, and there were three. Every measurement was correct, and the composition of correct measurements was a system that had just destroyed two hours of two signals without emitting a single error.

That happens whenever the thing measured is a proxy for the thing you care about. Processes are a proxy for replicas; acceptance is a proxy for durability; recovery of ingestion is a proxy for recovery of the platform. Each proxy is cheap to measure and right almost all the time, which is exactly why nobody notices the day it comes apart.

The defence is to measure the property itself at least once. Ask the ring how many owners a shard has rather than counting processes. Compare what was accepted against what can be read back rather than trusting the acknowledgement. And kill a node on purpose, on a schedule — an untested high-availability claim is not a weaker claim than a tested one, it is a different kind of statement, and it belongs in a design document rather than a runbook.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Three instances, one copy

2. The lost window is two hours because that is the WAL’s job

3. The three loud symptoms are the system working

Resolution

Verification

Prevention