Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

advancedloki-labels~30 min

Break/Fix: Loki High-Cardinality Labels

Reported symptoms

  • ●Loki memory has spiked

Evidence

  • · loki_ingester_streams climbs
Diagnosis and resolutionclick to reveal

Root cause

A new high-cardinality label was added.

Remediation

1. Identify the stream. 2. Fix the label set. 3. Migrate to structured metadata.

Verification

Memory returns to baseline.

Prevention

Loki label audit

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Loki memory has spiked

Available evidence:

  • loki_ingester_streams climbs

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Identify the stream. 2. Fix the label set. 3. Migrate to structured metadata.

Verification

Memory returns to baseline.

Rollback

Revert

Prevention

Loki label audit