Break/Fix: Alert Not Firing

Scenario

You are operating a production observability platform. The following symptoms appear:

• Production is broken
• No alert fired

Available evidence:

• /api/v1/rules shows the rule
• The condition is true

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

1. Form hypothesis, find evidence, test, validate.
2. Use the available evidence above to bound the search.
3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

1. Identify the failing component.
2. Apply the remediation pathway.
3. Validate with the verification step.
4. Document the incident.

Remediation

1. Test the rule. 2. Check Alertmanager routing. 3. Check receiver. 4. Fix.

Verification

Alert fires and reaches receiver.

Rollback

Revert rule / route change

Prevention

Test alerts during onboarding; canary alerts in production.