Loki Loss

A log shipper page fires: a payment-service fleet has been silent for ten minutes, and Grafana returns no results. The Loki query service is up; the ready endpoint is green; but every LogQL query against the last hour returns zero lines. You open the ingester logs and find a stream of tsdb index error messages about missing files. The object store bucket is gone.

This lesson is the recovery procedure for that incident. The procedure depends on which piece of Loki was lost.

What it is

Loki loss is any condition in which LogQL queries cannot return the truth. Three shapes:

• Ingester loss. The in-memory per-tenant state is gone, and the WAL on local disk is missing or unreadable. Unflushed chunks — usually the last 0 to 30 minutes — are lost. Older chunks live in the object store.
• Object-store loss. The bucket holding chunks and the modern TSDB index is gone. Loki cannot read anything older than the last successful flush.
• Index drift. The compactor was stopped, misconfigured, or left behind a half-built index. Queries return partial results for the affected time window.

The first job is to identify which shape you have. The recovery procedure for each is different.

Why a sysadmin cares

Loki is the first stop in an incident that does not show up in metrics. A memory leak that does not move RSS, a slow database query that does not change latency, a feature flag that does not affect throughput — none of these generate the metric signature that alerts on. The first signal is in the logs, and a platform whose logs are missing is a platform whose incidents are uninvestigable.

The recovery procedure also has a sharp edge: Loki 3.x ships the compactor as the only component that deletes anything. A misconfigured retention applied during an incident becomes a permanent deletion. Recovery is not just bring it back; it is bring it back without deleting the truth.

How it works

Loki is a write path, a read path, and a compactor:

   distributor (auth, rate limit)
        |
        v
   ingester  --->  WAL on local disk
        |
        v
   chunk upload ----> object store
                              |
                              v
                       TSDB index
                              ^
                              |
   compactor (retention, dedup, index merge)
                              ^
                              |
   querier / query-frontend --- LogQL ---> object store

The ingester buffers incoming streams and flushes them as per-tenant chunks on a configurable interval. The WAL — when enabled — captures every series assignment before the chunk is built, so an ingester restart can replay the unflushed window.

The object store is canonical. Chunks and (in the modern schema) the index live there. Object-store loss is a stack-wide event.

Under the hood

How to configure it

A Loki that is built to be rebuilt has the WAL enabled, the compactor sized for the index size, and the object store replicated.

# /etc/loki/loki.yaml
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    s3: s3://eu-west-1-loki-chunks
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    shared_store_key: index

ingester:
  wal_enabled: true
  wal_dir: /loki/wal
  wal_replay_memory_on_shutdown: true
  chunk_idle_period: 30m
  max_chunk_age: 2h
  chunk_encoding: snappy

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: s3

limits_config:
  retention_period: 744h
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 64

The WAL directory must be on a separate volume from the index directory. A volume loss that takes the WAL must not take the object store, or vice versa.

How to validate it

Four checks, in order, before declaring recovery complete:

# 1. The ingester has finished WAL replay and is ready.
curl -sf http://loki:3100/ready | head -1
# expected: ready

# 2. The distributor sees all ingesters.
curl -s http://loki:3100/distributor/ring \
  | jq '.instances | map({id: .id, state: .state})'

[
  { "id": "loki-ingester-0", "state": "ACTIVE" },
  { "id": "loki-ingester-1", "state": "ACTIVE" }
]

# 3. The compactor has caught up.
curl -s http://loki-compactor:3100/status | jq .
# expected: tables are not in compacting state

# 4. A canary LogQL query returns expected results.
curl -s -u "${LOKI_USER}:${LOKI_PASS}" \
  -G http://loki:3100/loki/api/v1/query \
  --data-urlencode 'query={job="loki"} |= "wal replayed"' \
  --data-urlencode 'limit=10' \
  | jq '.data.result | length'

If any check returns empty or unhealthy, the recovery is not done.

How it can fail

• WAL replay hangs because the WAL directory has a corrupt segment. The ingester logs a CRC error and refuses to start. Removing the corrupt segment recovers the ingester but loses every series in that segment.
• Ingesters come up empty after a network partition because the ring gossip failed to converge. Distributors write to a stale ring and the new ingesters receive nothing.
• Object store bucket restored from a stale snapshot. The bucket is back, but it is missing the last N hours of writes that were not in the snapshot. The RPO is the snapshot age.
• Compactor restarted while in the middle of a compaction cycle. It leaves a half-built index prefix and refuses to start on next launch. Manual cleanup of the prefix is required.
• Retention applied to a tenant that should have been exempt. The compactor deletes the truth during what was supposed to be a recovery.
• Per-tenant rate limit reset during recovery. Ingest flood overruns the distributor and the cluster enters a back-pressure spiral that looks like a second incident.

How to troubleshoot it

The diagnostic order:

1. Is the ingester running? (kubectl get pods -l app=loki, component=ingester)
2. Is the WAL replay finished? (logs say replay completed or wal replayed)
3. Is the distributor seeing all ingesters? (/distributor/ring)
4. Is the object store reachable? (/ready on the ingester and /compactor/status)
5. Are queries returning results? (/loki/api/v1/query)
6. Is the compactor caught up? (/compactor/status)

Security implications

The object store holds every log line the platform has retained. A lost or leaked bucket is a compliance event, not just a durability event. Encryption at rest is mandatory, and the IAM policy on the bucket must scope to the Loki service account, not to a shared user.

The ingester WAL on local disk contains the same data, briefly, before it is flushed. A volume that escapes the host — a snapshot shipped to a less-secure bucket — carries the same exposure.

Performance implications

WAL replay scales with the unflushed window. A 30-minute WAL on a busy tenant takes minutes to replay on cold-start; a 4-hour WAL takes hours. The WAL is sized by the chunk_idle_period and the ingester flush interval; a longer flush interval means a longer WAL means a longer RTO.

Compactor cycles scale with the index size. A platform whose index has grown without a working compactor has a multi-day recovery; the first compaction pass is the bottleneck.

Production guidance

• Keep the WAL on a dedicated volume with its own backup. The WAL is the only place the unflushed window exists.
• Run the compactor as a singleton with a leader election. A compactor that runs twice corrupts the index.
• Test the recovery against a non-production Loki that points at a separate bucket. Production recovery is not a place to find out the credentials are wrong.

Verification

You should now be able to answer:

• What three loss shapes does a Loki recovery distinguish, and what is the right first action for each?
• Why does the WAL need its own dedicated volume?
• What is the danger of running the compactor against a half-restored bucket?
• Which four endpoints confirm a Loki cluster is healthy after recovery?

Quiz