Scenario
You are operating a production observability platform. The following symptoms appear:
- Loki has no logs from a service
Available evidence:
- Loki /metrics shows ingestion 0 for the stream
Your task
Determine the cause, recover, document, and validate.
Investigation
The investigation follows the discipline taught in Part XCVIII:
- Form hypothesis, find evidence, test, validate.
- Use the available evidence above to bound the search.
- Reach one of the likely root causes.
Recovery procedure
(Do not reveal until you have reasoned through the problem.)
- Identify the failing component.
- Apply the remediation pathway.
- Validate with the verification step.
- Document the incident.
Remediation
- Check collector logs. 2. Verify configuration. 3. Check network. 4. Fix.
Verification
Logs appear in Loki; logcli query returns expected lines.
Rollback
Revert
Prevention
Test with a synthetic log entry