Scenario
You are operating a production observability platform. The following symptoms appear:
- Alert fires; the underlying condition does not clear
Available evidence:
- Alert has been firing for hours or days
Your task
Determine the cause, recover, document, and validate.
Investigation
The investigation follows the discipline taught in Part XCVIII:
- Form hypothesis, find evidence, test, validate.
- Use the available evidence above to bound the search.
- Reach one of the likely root causes.
Recovery procedure
(Do not reveal until you have reasoned through the problem.)
- Identify the failing component.
- Apply the remediation pathway.
- Validate with the verification step.
- Document the incident.
Remediation
- Verify the condition is still true. 2. If true: treat as incident. 3. If false: check
for:duration and resolve.
Verification
Alert transitions through firing → resolved.
Rollback
Revert rule change
Prevention
Rules should test the recovery path too, not only the firing path.