Scenario
You are operating a production observability platform. The following symptoms appear:
- Loki ingest rate has spiked 20x
Available evidence:
- A new debug-level stream was enabled
Your task
Determine the cause, recover, document, and validate.
Investigation
The investigation follows the discipline taught in Part XCVIII:
- Form hypothesis, find evidence, test, validate.
- Use the available evidence above to bound the search.
- Reach one of the likely root causes.
Recovery procedure
(Do not reveal until you have reasoned through the problem.)
- Identify the failing component.
- Apply the remediation pathway.
- Validate with the verification step.
- Document the incident.
Remediation
- Identify the stream. 2. Revert log level. 3. Validate.
Verification
Ingest returns to baseline.
Rollback
Revert
Prevention
Log rate alerts