Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediatealertmanager~25 min

Break/Fix: Alertmanager Routes Wrong

Reported symptoms

  • ●Alerts go to wrong receiver

Evidence

  • · amtool alert show shows wrong route
Diagnosis and resolutionclick to reveal

Root cause

The routing tree is misconfigured.

Remediation

1. Inspect amtool config show. 2. Fix the routing matchers. 3. Test with a synthetic alert.

Verification

Synthetic alert reaches the correct receiver.

Prevention

Test routing before changes

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Alerts go to wrong receiver

Available evidence:

  • amtool alert show shows wrong route

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Inspect amtool config show. 2. Fix the routing matchers. 3. Test with a synthetic alert.

Verification

Synthetic alert reaches the correct receiver.

Rollback

Revert routing change

Prevention

Test routing before changes