Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateloki-storage~25 min

Break/Fix: Loki Storage Unavailable

Reported symptoms

  • ●Loki cannot write to object store

Evidence

  • · loki_ingester_chunks_dropped total rises
Diagnosis and resolutionclick to reveal

Root cause

The object store is down.

Remediation

1. Verify object store. 2. Apply retention_compactor_interval_block_size or failover.

Verification

Ingestion resumes.

Prevention

Failover / multi-region

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Loki cannot write to object store

Available evidence:

  • loki_ingester_chunks_dropped total rises

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Verify object store. 2. Apply retention_compactor_interval_block_size or failover.

Verification

Ingestion resumes.

Rollback

Revert

Prevention

Failover / multi-region