Break/Fix: Loki Storage Unavailable · Break/Fix · Observability

Reported symptoms

The page at 23:20 is for ingester memory. All three Loki ingesters have been climbing in lockstep since 21:38. There has been no deploy, and ingest volume has not changed.

The on-call finds four more things wrong, none of which look related:

The write-ahead log volumes are filling. All three are at 61 percent and gaining about 4 GiB an hour. They normally sit flat at around 8 percent.
Grafana panel snapshots have stopped working. The render succeeds and the upload fails. A ticket was raised at 22:10 and assigned to the Grafana owner.
The nightly platform backup failed. It reported an upload error at 23:00. Nobody has looked at it yet.
Tempo is buffering too. Nobody noticed until somebody thought to check.

And Loki looks completely healthy. A spot check over the last three hours returns exactly what the producers say they sent. A spot check over yesterday does too. Queries are fast. Nothing has been rejected. No producer has seen an error.

The first responder checks the storage cluster, because ingester memory and storage are related in the runbook. The MinIO console reports ONLINE. Capacity is 43 percent used. They list the bucket from their workstation and download a chunk, which works. They write “storage is fine” in the incident channel and go back to the memory graph.

There is a change ticket open: routine kernel patching of the storage nodes, approved, applied node by node.

Evidence provided

Read-only / Safenothing has been flushed since 21:38

$ promtool query instant http://prometheus:9090 'sum(rate(loki_ingester_chunks_flushed_total[5m]))'

=> 0    (baseline for this hour: about 210/s)

Illustrative output

Read-only / Safeingest is completely normal - everything is arriving and nothing is leaving

$ promtool query instant http://prometheus:9090 'sum(rate(loki_distributor_bytes_received_total[5m]))'

=> 1.09e+07   (30-day baseline: 1.11e+07)

Illustrative output

Read-only / Safereads succeed; the PUT 5xx counter is the only one still moving

$ curl -s http://loki-ingester-0.loki:3100/metrics | grep loki_objstore_request_duration_seconds_count

loki_objstore_request_duration_seconds_count{operation="GET",status_code="200"} 8841203
loki_objstore_request_duration_seconds_count{operation="LIST",status_code="200"} 194402
loki_objstore_request_duration_seconds_count{operation="PUT",status_code="200"} 2210984
loki_objstore_request_duration_seconds_count{operation="PUT",status_code="5xx"} 61147

Illustrative output

Read-only / Safethis number is the deadline

$ promtool query instant http://prometheus:9090 'node_filesystem_avail_bytes{mountpoint="/var/loki/wal"}'

loki-ingester-0  1.44e+11
loki-ingester-1  1.41e+11
loki-ingester-2  1.46e+11

growth: about 4 GiB/h per ingester  ->  roughly 95 minutes of headroom

Illustrative output

Read-only / Safehalf the drives are gone, and the cluster still calls itself online

$ mc admin info storage

4 servers online, 8 drives online, 8 drives offline
minio-a  4 drives online
minio-b  0 drives online   (drained)
minio-c  4 drives online
minio-d  0 drives online   (drained)

Illustrative output

Read-only / Safelisting works, which is what convinced everyone storage was healthy

$ mc ls storage/loki-chunks/ | head -3

[2026-08-18 20:11:04 UTC]  1.4MiB  01JB...c7/
[2026-08-18 20:11:09 UTC]  980KiB  01JB...d1/
[2026-08-18 20:11:12 UTC]  1.2MiB  01JB...d9/

Illustrative output

Read-only / Safeone byte, and the whole incident collapses into one line

$ printf x > /tmp/probe.txt && mc cp /tmp/probe.txt storage/loki-chunks/probe.txt

copy failed

Illustrative output

And from the maintenance automation:

kernel-patch  group: rack-b   hosts: minio-b, minio-d, app-14, app-15, app-16
              started 21:36   mode: drain, patch, return - one host at a time
              note: hosts within a group are drained together

Work the evidence before reading on

Six questions, in the order the evidence answers them.

Bytes received is normal and chunks flushed is zero. What is Loki doing with the difference, and where is it putting it?
Every query returns correct data, including queries over the last hour. If nothing has been written to the object store since 21:38, where is that data being served from?
Reads against the bucket succeed and writes fail. What kind of storage fault distinguishes between the two, and why would a health check never catch it?
Half the drives are offline and the cluster calls itself online. Is that a bug, or is it a precise statement about something?
The snapshot upload, the backup job and the Tempo flush were each raised to a different team. What do those three have in common with each other and with Loki?
The write-ahead log volume has about 95 minutes of headroom. What happens at the end of those 95 minutes, and how does that change what you do in the first five?

Before continuing: the first responder listed the bucket and downloaded an object, and concluded storage was healthy. What question did that test answer, and what question was the incident actually about?

Root cause

Reads and writes do not need the same number of drives

The object store is an on-premises MinIO cluster: four nodes, four drives each, arranged as a single erasure set of sixteen.

An erasure-coded store splits each object into data and parity shards spread across the drives. Reading an object that already exists needs only enough shards to reconstruct it. Writing a new object needs enough drives available to place a complete, correctly parity-protected set of shards - which is more. Those two thresholds are different by design, and the gap between them is exactly where this incident lives.

With eight of sixteen drives offline, the cluster sits in that gap. It can reconstruct everything it already holds and cannot store anything new. It is simultaneously perfectly healthy and completely unavailable, and which one you observe depends entirely on which verb you test.

Every check anyone ran tested the read verb. The console said ONLINE, which is true. Capacity was 43 percent used, which is also true and completely irrelevant - this is not a capacity failure and the free-space graph will never show it. A listing worked. A download worked. The one test that would have ended the investigation in its first minute was a one-byte upload, and it took an hour for anyone to try it.

Two nodes went down together because of how the automation is grouped

The kernel-patching automation drains one host at a time within a group, which is correct and careful. Its groups are defined by rack, which is a reasonable way to group hosts, because racks are how power and switching fail.

The storage cluster’s failure domain is not the rack. It is the erasure set, and two of the four storage nodes happen to be racked together. So an automation working exactly as designed took out half of one erasure set, and nothing anywhere in the pipeline knew that erasure sets existed.

Why nothing broke visibly for two hours

Loki’s response to a failed flush is to keep the chunk in memory, keep the record in the write-ahead log, and keep accepting writes. That is the correct behaviour: a transient storage failure should not become an ingestion failure, and the write-ahead log exists precisely so that the data survives while the durable write path is unavailable.

The consequence is that a total loss of durable storage is invisible from almost every angle:

Observation	Why it looks fine
Producers see no errors	Distributor and ingester both accept and acknowledge the write
Recent queries are correct	Queriers read from ingester memory for data not yet flushed
Older queries are correct	Those chunks were written before 21:38 and read fine
Ingest dashboards are green	Bytes received never changed
Storage capacity is normal	This is not a capacity failure

What is not fine is that for two hours the platform durably stored nothing, and the only thing standing between a storage outage and permanent data loss is a write-ahead log volume with ninety-five minutes left on it.

The other three tickets are this ticket

The Grafana snapshot upload, the nightly backup and Tempo’s flushes all write to the same bucket. Each failed at 21:38 for the same reason, and each was raised separately, to a different team, as its own small problem.

That fan-out is the most useful pattern in the incident. Several unrelated systems failing in the same minute is rarely a coincidence, and the thing to look for is not what they do but what they depend on.

Under the hood

The write-ahead log is the reason this is a delay rather than a loss, and understanding its limit is what turns panic into a schedule.

Every record an ingester accepts is appended to a write-ahead log on local disk before it is acknowledged. Chunks accumulate in memory and are flushed to the object store when they fill, when they reach their maximum age, or when the stream goes idle. Once a chunk is durably in the object store its records are no longer needed in the log, and the log is truncated. When flushes fail, that truncation stops, and the log grows at roughly the ingest rate.

So the outage has a computable end. Free bytes on the write-ahead log volume, divided by the ingest rate, is the time until the ingester can no longer append records - and an ingester that cannot append cannot acknowledge, at which point producers start seeing errors and data begins to be lost for real.

That single number reframes the whole incident. It tells you whether you have time to fix the cause properly or must start shedding load; it tells you whether a twenty-minute repair is acceptable; and it converts “how bad is this?” into a decision with a clock on it. It is also why deleting write-ahead log segments to free space is the one genuinely destructive action available: the segments are the data.

Restarting the ingesters is the reflex, and it makes the deadline shorter

Ingester memory is climbing, so the reflex is to restart the ingesters.

An ingester replays its write-ahead log on startup, rebuilding exactly the in-memory chunks that were there before, and then tries to flush them into the same object store that is still refusing writes. The end state is identical to the start state, with the replay time spent and the log no smaller.

The cost is worse than nothing. Replay takes minutes, during which that ingester is not accepting writes, so the other two carry the whole load and their own logs grow faster. Restarting all three in sequence under this fault is a reliable way to turn ninety-five minutes of headroom into sixty.

The general rule: when a component is buffering because a downstream dependency is refusing work, restarting the buffer never helps. It only discards or replays the buffer, and one of those is worse than the other.

Resolution

Compute the deadline first, before deciding anything else. Free space on the write-ahead log volume divided by its growth rate is the time until data loss becomes real, and every other decision in this incident depends on whether that number is 95 minutes or 20.
Run the write test. Copy a one-byte object into the bucket. Success and failure mean entirely different investigations, and no amount of listing, downloading or console-reading substitutes for it.
Restore write quorum by the cheapest route. Only one of the two drained nodes has to come back to cross the threshold, so aborting the patch on one node and returning it is almost always faster than completing the work on the other. Confirm with the drive count, not with the cluster status.
Pause the maintenance automation before it moves on to another host. It is working as designed and its design does not know about erasure sets; the next host it drains could be in the other rack.
Do not restart the ingesters. Replay rebuilds the same unflushable state, spends the replay time, and drops write capacity while it runs.
Do not delete write-ahead log segments to reclaim space. That is the data you are trying to save, and deleting it produces the exact outcome the buffer exists to prevent.
Do not raise the chunk idle period or maximum chunk age to reduce flush attempts. Holding chunks longer increases the memory that is already the presenting symptom.
If the deadline is genuinely short, grow the write-ahead log volume online if the storage layer allows it, or shed the lowest-value streams at the agent. Shedding is a deliberate, recorded, bounded data loss chosen over an undirected one, and it needs a name against it in the incident log.
If the storage team can return a node well inside the deadline, hold - and write the hold down. Staffing the incident, stopping other changes and setting an explicit time at which shedding begins is a decision; quietly waiting is not.

Verification

The write verb works, tested directly. A one-byte object copies in and is removed again. The read path never stopped working and will keep reassuring you either way.
Loki is flushing rather than merely able to flush. rate(loki_ingester_chunks_flushed_total[5m]) is non-zero on all three ingesters, and the PUT series of loki_objstore_request_duration_seconds_count is accumulating against a 2xx status code with the 5xx series flat.
The backlog is draining, not just stable. loki_ingester_memory_chunks falls, loki_ingester_wal_bytes falls, and the write-ahead log volume returns to its usual few percent. If memory falls and the log does not, something is still failing quietly.
The gap is measured rather than assumed. Query the incident window and reconcile line counts against what the producers say they sent. That the write-ahead log held is a hypothesis until somebody checks, and this is the only week anyone will remember to.
The drive count, not the cluster status, confirms the storage side. The status field said ONLINE throughout an outage in which nothing could be written, so it is not evidence of anything.
The collateral tickets are closed out in the same pass: the backup job re-runs successfully, a panel snapshot uploads, and Tempo has drained its own buffer.
The maintenance automation now refuses to drain two nodes of the same erasure set together. Test it by asking it to, in a window, and watching it decline. A guardrail that has never refused anything has not been tested.
The new alerts fire on a rehearsal. Take one storage node out deliberately and confirm the flushes-stopped alert and the write-ahead-log-time-remaining alert behave as intended while reads stay green throughout.

Prevention

Alert on flushes stopping, not on storage being down. Chunks flushed falling to zero while bytes received stays non-zero is a precise and unambiguous statement that the durable write path has failed, and it fires in minutes. It is also the alert nobody writes, because the intuitive thing to monitor is the storage system - and the storage system spent this entire incident reporting itself healthy.
Express write-ahead log headroom as time remaining. Sixty-one percent sounds comfortable and ninety-five minutes does not, and only one of those two numbers supports a decision at 23:20.
Make the storage health check exercise the operation you depend on. A synthetic probe that writes a small object, reads it back and deletes it tests all three verbs. A read-only check passes cleanly through a total write outage, which is what happened here.
Encode the storage layer’s failure domain in the automation that drains hosts. Grouping by rack is right for power and switching and wrong for an erasure set. A constraint that exists only in a wiki page is not a constraint; it has to live in the thing that would otherwise violate it.
Keep the write-ahead log on a volume that can be grown online. Its size is, quite literally, the length of storage outage the platform can survive without losing data. Treat that as a capacity decision rather than a default.
Do not let the alerting path depend on the failing component. The Loki ruler persists its state to the same bucket, so an alert about the bucket may be evaluated by something that also cannot write to it. Alerts about the logging platform belong somewhere the logging platform cannot take with it.
Read a fan-out of small tickets as one signal. A snapshot upload, a backup job and a trace flush all failing in the same minute is not three problems. Ask what they share before asking what each of them does.

What if the write-ahead log volume had been half the size?

Then the deadline would have been about fifty minutes instead of ninety-five, and almost every decision in this incident would have been different.

There would have been no time to work out that the storage cluster was the cause, because the first hour went on the memory graph and the console status. The team would have arrived at the write test at roughly the moment the log filled. Producers would have started receiving errors, the collectors would have begun spooling and then dropping, and the incident would have converted from a recoverable delay into a permanent gap in the logs covering exactly the window the postmortem needs.

The uncomfortable part is that the write-ahead log size was never chosen with this in mind. It was whatever the volume template specified, and it happened to be generous. The platform survived a two-hour storage outage without losing a line because of a number nobody had reasoned about.

The generalisation is worth carrying: every buffer in a pipeline is a statement about how long its downstream dependency is allowed to be unavailable, whether or not anyone has read it that way. Agent spool disks, ingester write-ahead logs, queue depths - each of them has a duration hidden inside it. Working those durations out in advance, and writing them next to the component, is far cheaper than discovering them during an incident with a clock running.

Why this matters

For two hours a logging platform accepted every line it was sent, answered every query correctly, reported itself healthy on every dashboard, and wrote nothing at all to durable storage. The only thing preventing a two-hour hole in the logs was a local disk that happened to be large enough.

Two things are worth carrying out of this. The first is that “is it up?” is not a question with an answer. Storage has verbs, and a system can be perfectly available for one and completely unavailable for another. Read quorum and write quorum are different numbers in every erasure-coded store, and the same shape recurs elsewhere - a database that serves reads from a replica while the primary is gone, a registry that serves pulls and refuses pushes, a filesystem remounted read-only under you. Any check that exercises only the cheap verb will pass through the whole outage.

The second is about buffers. Loki behaved correctly throughout: it protected the producers, it protected the data, and it bought the operators an hour and a half they did not know they had. But a buffer converts a loud failure into a silent one, and a silent failure with a deadline is the most dangerous shape an incident can take. Wherever something buffers on your behalf, find out how long it can do so, and make sure the number is on a dashboard before you need it.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

Reads and writes do not need the same number of drives

Two nodes went down together because of how the automation is grouped

Why nothing broke visibly for two hours

The other three tickets are this ticket

Resolution

Verification

Prevention