Runbook: Recover Loki

“Recover Loki” is not one procedure. It is three, and they are distinguished by which of Loki’s three durability layers was lost. The layers fail independently, present almost identically to a user, and have first moves that are actively harmful when applied to the wrong one.

   distributor            <- auth, rate limit, no durable state
        |
        v
   ingester  --->  WAL on local disk     <- unflushed window, minutes
        |
        v
   chunks + TSDB index ---> object store <- canonical, everything older
                                    ^
                                    |
                              compactor  <- the ONLY component
                                             that deletes anything

Read that diagram once more before touching anything. The compactor sits at the bottom with delete authority over the canonical store, and almost every way this runbook can turn a recoverable incident into a permanent one runs through it.

Classify first: which layer was lost

Shape	What is gone	What it looks like	First move
INGESTER	In-memory streams, and the WAL that would replay them	Recent lines missing; historical queries fine; ingester restarting or refusing to start	Read the ingester log, then let replay finish
OBJECT STORE	Chunks and the index	Historical queries empty; the last ~30 minutes still answer from ingester memory	Stop the compactor, then verify the bucket
INDEX / COMPACTOR	A coherent index for some window	Partial results over one window; everything else fine	Stop the compactor, then inspect the prefix

The middle row hides a trap worth naming. When the bucket is gone, recent queries keep working, because the querier serves the recent window from ingester memory for query_ingester_within - thirty minutes by default. For that half hour the platform looks healthy to anyone who checks “is Loki up” with a five-minute query. The failure surfaces later, and it surfaces as history, which is exactly when people trust it most.

When this runbook applies, and when it does not

It applies when Loki cannot return the truth: lines that were ingested are unavailable, or lines that should be ingestible are not being accepted, because a durability layer has been lost or damaged.

It does not apply when:

Loki is healthy and the logs never arrived. If the distributor never received the bytes, there is nothing to recover here; that is the ingestion path, and it starts on the emitting host.
The query is wrong. An empty panel against a healthy cluster is a query problem, and running a recovery procedure against it will make things worse for no reason.
The window is outside retention. The compactor deleted it by policy. That is a retention decision to revisit, not an incident to recover.
Loki is merely slow. Saturation is a capacity and query-shape problem. Restarting components under load turns a slow platform into an unavailable one.

Blast radius

Action	Reversible?	What it costs if wrong
Reading `/ready`, `/config`, ring and logs	n/a	Nothing
Stopping the compactor	Yes - start it again	Retention pauses; index grows. Cheap
Restarting an ingester	Partly	Replay time; and any in-memory state not in the WAL
Moving a WAL segment aside	No	Every series in that segment, permanently
Pointing Loki at a restored bucket	Depends	A stale snapshot silently sets your RPO
Running the compactor mid-restore	No	Retention applied to gaps. Permanent deletion
Raising limits to absorb a replay flood	Yes	Back-pressure spiral if the flood is larger than the platform

Two rows are one-way doors. Both are listed in the escalation path because neither should be walked through by one tired operator alone.

Step 1 - Classify, without changing anything

Read-only / Safethe three reads that name the loss shape

LOKI=http://loki.monitoring.svc:3100

curl -s "$LOKI/ready"

curl -s "$LOKI/distributor/ring" | jq '.instances | map({id: .id, state: .state})'

kubectl logs -n loki -l component=ingester --tail=200 | grep -iE 'wal|replay|crc|error' | tail -20

ready

[
{ "id": "loki-ingester-0", "state": "ACTIVE" },
{ "id": "loki-ingester-1", "state": "JOINING" }
]

Illustrative output

An ingester stuck in JOINING is usually replaying its WAL, and that is normal behaviour rather than a fault. /ready is component-aware: the ingester is not ready until replay finishes, the compactor is not ready until it acquires its ring lock, and the querier is not ready until it has loaded index files. A check that accepts any HTTP 200 as healthy will miss all three.

Step 2 - Stop the compactor, before anything touches the bucket

If the object store is implicated in any way - restored, repointed, partially available, or merely suspected - the compactor stops first.

Service impact possiblethe cheapest irreversibility control in this runbook

kubectl scale -n loki deployment/loki-compactor --replicas=0

kubectl get pods -n loki -l component=compactor

Stopping the compactor pauses retention and lets the index grow. That is the entire cost, and it is trivially reversible. Leaving it running against a bucket that is mid-restore lets it apply retention to a window that only looks old because the objects are not back yet.

Step 3 - INGESTER: replay, and the segment you may have to sacrifice

Let replay finish. Replay time scales with the unflushed window, which is set by chunk_idle_period and the flush behaviour: a longer flush interval means a longer WAL, which means a longer recovery. That relationship is worth remembering at configuration-review time, because it is where your RTO is actually decided.

If the ingester refuses to start and logs a CRC error against a WAL segment, you are at the one-way door.

Data-loss riskmove it aside - never delete it

# Second operator present, and the window this segment covers recorded first.
WAL_DIR=/loki/wal
SEGMENT=00000042

mv "$WAL_DIR/$SEGMENT" /var/tmp/loki-wal-quarantine-$SEGMENT

Moving the segment lets the ingester start with a truncated WAL. Every series in that segment is gone. Keep the file: it is the only artefact that can later answer “what was in the gap”, and deleting it converts an accountable loss into an unexplainable one.

Count series before and after the restart, decide whether the gap fits inside the stated RPO, and escalate if it does not.

Step 4 - OBJECT STORE: verify before you repoint

A restored bucket is not a recovered bucket. Verify that the objects covering the affected window are actually present, under the prefix and in the region Loki is configured to use.

Read-only / Safedoes the restored bucket contain the window you need?

BUCKET=prod-loki-chunks
TENANT=prod

aws s3api list-objects-v2 --bucket "$BUCKET" \
--prefix "index/$TENANT/" --max-items 20 \
--query 'KeyCount' --output text

Two failure shapes are worth recognising by their signature. Every request returning 301 PermanentRedirect is a region mismatch, not a permissions problem. Reads succeeding while compactor operations are denied is an IAM policy that was written against the read path only - which passes every check you run today and fails at the next retention cycle.

Restore into a separate prefix or bucket and verify there, whenever the object store offers that option. Restoring in place over newer objects is not reversible.

Step 5 - Bring it back in dependency order

Object store reachable, then ingesters, then queriers, then the compactor last. Deploy configuration from the version-controlled commit rather than hand-editing the running config, then prove the two agree.

Read-only / Safevalidate the config before the binary, and the ring after it

loki -config.file=/etc/loki/loki.yaml -verify-config

curl -s "$LOKI/config" | head -40

curl -s "$LOKI/runtime-config" | jq '.overrides | keys'

Step 6 - Prove it end to end, then find the gap

Readiness proves processes started. Only a round trip proves the platform works.

Read-only / Safepush a known marker, then find it again

MARKER="recovery-$(date -u +%s)"
NOW_NS="$(date +%s)000000000"

curl -sf -X POST "$LOKI/loki/api/v1/push" \
-H 'Content-Type: application/json' \
-d '{"streams":[{"stream":{"job":"smoke"},"values":[["'"$NOW_NS"'","'"$MARKER"'"]]}]}'

sleep 5

export LOKI_ADDR="$LOKI"
logcli query --since=5m --limit=5 --quiet '{job="smoke"}'

Then run one query over a window older than query_ingester_within. Recent queries are served from ingester memory and will succeed even when the object store path is broken; a historical query is the only one that exercises the bucket.

Finally, establish the gap. Start, end, tenants, and lost versus merely delayed. A recovered Loki whose gap is undocumented is worse than an obviously broken one, because the next incident will read that empty window as evidence that nothing happened.

Hold, when the queue is draining

If collectors buffered during the outage and are now replaying, the correct action is frequently to watch rather than to act. Lines are arriving late rather than not at all, and a restart during the drain converts late into lost.

Give the hold the same shape as any other action: an owner by name, a checkpoint (ring state and distributor byte rate re-read every five minutes), and an exit condition (the backlog drains, or rejections start climbing, at which point the replay flood is larger than the platform and limits become a real decision).

Common patterns

What you see	Shape	Where to look
Recent lines missing, history fine	INGESTER	WAL replay progress in the ingester log
Ingester will not start, CRC error	INGESTER	The named WAL segment - this is the one-way door
History empty, last 30 min fine	OBJECT STORE	Bucket contents, then `query_ingester_within`
Every bucket request 301	OBJECT STORE	Region mismatch, not permissions
Reads fine, retention never advances	COMPACTOR	IAM policy missing the compactor’s actions
Partial results over one window	INDEX	Half-built index prefix from a killed compaction
Compactor will not start after a kill	INDEX	Move the partial prefix aside, do not delete
Pushes accepted, nothing queryable	RING	`/distributor/ring` - a stale ring accepts and discards
Everything ready, overrides ignored	CONFIG	`/runtime-config`, not the file on disk
Recovery looks fine, gap unknown	PROCESS	Nobody established the boundaries

Prevention

Three controls decide how this incident goes before it starts.

The first is the WAL on its own volume, backed up on its own schedule. It is the only place the unflushed window exists, and a volume loss that takes both the WAL and the object store turns a bounded gap into an unbounded one.

The second is a compactor that runs as a singleton with leader election. Two compactors racing corrupt the index, which produces the partial-results shape that is the hardest of the three to diagnose.

The third is rehearsing this procedure against a non-production Loki pointed at a separate bucket. Production recovery is a poor place to discover that the credentials in the manifest have never been used for a write. Every step in this runbook is cheap to practise and expensive to learn during an incident.

Why this matters

The reason this runbook spends its first two steps on classification and on stopping a component is that Loki recovery has an unusual asymmetry: the diagnosis is cheap and almost entirely read-only, while two of the available actions are permanent. Moving a WAL segment aside deletes a window of logs. Running the compactor against a half-restored bucket deletes considerably more, silently, and with the full confidence of a component doing exactly what it was configured to do.

The second thing worth internalising is the thirty-minute lie. When the object store is gone, Loki keeps answering recent queries out of ingester memory, so every quick health check passes and the platform reports itself healthy. The failure is real and it is already happening; it is simply not visible yet in the window an operator naturally checks. That is why the verification list above insists on a historical query. A recovery confirmed only against recent data has confirmed the one thing that was never broken.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation