Inhibit Rule Review

Inhibit rules are living configuration. They describe the relation between alerts at a moment in time. The moment the team migrates the checkout service off Postgres, the rule that says “Postgres down suppresses checkout-svc” becomes a silent failure: the rule still loads, but it never fires against the migrated checkout service. Six months later a Postgres outage hits and the checkout team is paged because the inhibit rule no longer covers them.

The same drift happens in the opposite direction. A rule written for a single-cluster platform is never updated when the team adds a second cluster. The rule fires against the original cluster and silently suppresses nothing on the new cluster. The new cluster’s on-call rota is paged for every host-down event.

A team that updates the dependency map quarterly keeps inhibition useful. A team that updates it never has stuck inhibit rules that either suppress the wrong alerts or nothing at all.

This lesson is the discipline of monthly review: the rubric, the test process, and the audit trail. The discipline is boring. The discipline is the difference between an observability stack that ages gracefully and one that silently masks real incidents.

What it is

The inhibit rule review is a recurring operational practice with three parts:

• The rubric. A fixed set of questions applied to every rule in the configuration. The questions are designed to catch stale rules, dead rules, and inert rules.
• The test. A repeatable procedure that proves the rule fires against a real or synthetic alert. The test is run after every change to the rule, and re-run during the monthly review.
• The audit trail. A record of every change to every rule: who changed it, when, why, and what the review found. The audit trail lives in the version control history of the configuration file and in a change log maintained alongside it.

Why a sysadmin cares

The failure shape is a platform that ages poorly. The inhibit rules were correct when written. The team added services, retired services, re-platformed services. The rules were not updated. The pager is either too loud (rules that do not fire) or too quiet (rules that suppress the wrong alerts). The team does not know which because nobody has looked.

The monthly review is a 90-minute meeting with a clear agenda. It produces a list of findings. The findings become change requests. The change requests become PRs. The PRs become an updated configuration. The discipline is mechanical. The discipline is the only thing that scales.

How it works

The rubric is a fixed set of questions. Apply them to every rule in the configuration. Document the answers.

Rule name :  HostDown-Suppresses-ServiceDown-Prod
Author    :  alice
Last review:  2026-07-15

Q1. Has the rule fired in the last 90 days?
    A: Yes (47 suppressions in 90d).

Q2. Does the source alertname still exist?
    A: Yes (HostDown is in the rules repo).

Q3. Does the target alertname still exist?
    A: Yes (ServiceDown is in the rules repo).

Q4. Are the equal labels still present on the source and
    target?
    A: Yes (instance, cluster).

Q5. Has the dependency map changed since the rule was last
    reviewed?
    A: No.

Q6. Has the cluster / region / env label vocabulary
    changed?
    A: No.

Q7. Is the dry-run still producing the expected
    suppression?
    A: Yes (re-tested 2026-07-15).

Decision:  retain.
Reviewer:  bob.
Date:      2026-08-15.

Each question is a check. A “no” answer is a finding. The finding becomes an action item. The action item becomes a change.

Finding categories

• Dead rule. Has not fired in 90 days AND the source alertname no longer exists. Action: remove the rule.
• Inert rule. Has not fired in 90 days AND the source alertname exists AND the equal clause is satisfiable. Investigate why; either the source is no longer expected to fire (action: keep and monitor), or the equal clause is unsatisfiable (action: fix or remove).
• Stale rule. Source and target exist, but the dependency map has changed. Action: update the labels to match the new map.
• Drifted label vocabulary. The cluster / region / env labels on the alerts no longer match what the rule expects. Action: update the rule to the new vocabulary, or fix the label injection at the Prometheus level.
• Over-broad rule. The dry-run shows the rule suppressing alerts it was not designed to suppress. Action: narrow the source or target matchers, or add to the equal clause.

The test

The monthly test is the same procedure used during the initial dry-run, applied to every rule. The test synthesises an alert, observes the suppression, and confirms the suppression matches the rule’s intent.

# 1. List every rule and its source alertname. amtool config
#    show prints the running configuration as YAML; the JSON
#    output carries the same YAML as a string, so the plain
#    form is the useful one here.
amtool config show --alertmanager.url=http://localhost:9093 \
  | sed -n '/^inhibit_rules:/,$p'

# 2. For each rule, post a synthetic source alert and a
#    matching synthetic target alert. Confirm the target is
#    suppressed.
amtool alert add alertname=HostDown \
  cluster="prod-eu" instance="prod-test-01" severity="critical"
amtool alert add alertname=ServiceDown \
  cluster="prod-eu" instance="prod-test-01" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | select(.labels.alertname=="ServiceDown")
         | {status: .status.state,
            inhibitedBy: .status.inhibitedBy}'

# 3. Confirm the suppression is for the expected reason.
#    Repeat for every rule.

# 4. Clean up the synthetic alerts. amtool has no command
#    that deletes an alert: alerts leave Alertmanager by
#    resolving. Re-post the same label set with a start and
#    an end in the past, or leave them to expire on their
#    own after the global resolve_timeout.
START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=HostDown \
  cluster="prod-eu" instance="prod-test-01" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  cluster="prod-eu" instance="prod-test-01" severity="warning" \
  --start="$START" --end="$END"

amtool alert query alertname=ServiceDown

The dry-run is the proof that the rule still does what it claims to do. A rule that fails the dry-run is a finding.

The audit trail

The audit trail is the version control history of alertmanager.yml plus a change log. The change log is a separate document that captures intent, not just diffs.

# changelog.yml (or a markdown file in the same repo)

- date: 2026-08-15
  reviewer: bob
  rule: HostDown-Suppresses-ServiceDown-Prod
  finding: retain
  notes: >
    Rule fired 47 times in 90 days. Source and target
    alertnames unchanged. Dependency map unchanged.
    Dry-run confirmed.

- date: 2026-08-15
  reviewer: bob
  rule: PostgresDown-Suppresses-CheckoutService
  finding: stale
  notes: >
    Checkout service migrated from postgres-prod to
    aurora-prod on 2026-06-12. depends_on label was
    updated to aurora-prod in the Prometheus rules.
    Inhibit rule source still references postgres-prod.
    Action: update source matcher to aurora-prod.

- date: 2026-08-15
  reviewer: bob
  rule: RegionalLBDown-Suppresses-Dependents
  finding: dead
  notes: >
    Source alertname RegionalLBDown no longer exists.
    The regional LB was retired in 2026-04. Action:
    remove rule.

- date: 2026-08-15
  reviewer: bob
  rule: AuthServiceDown-Suppresses-Dependents
  finding: retain
  notes: >
    Global rule. Source and target both present. Dry-run
    confirmed. No equal: clause is intentional and
    documented.

The change log captures the why behind every change. The git history captures the what. Together they form the audit trail that an incident reviewer can read six months later to understand the state of the inhibit rules.

Under the hood

How to configure it

There is no Alertmanager configuration for the review itself. The review is an operational practice; the artefacts are the change log and the git history. A typical setup:

# Repository layout for the alertmanager config

repo: observability-config
  alertmanager.yml       <- live configuration
  alertmanager-test.yml  <- test configuration for staging
  changelog/
    2026-Q3.yml          <- review findings for Q3
    2026-Q4.yml          <- review findings for Q4
  runbooks/
    inhibit-review.md    <- the rubric and the test procedure
  scripts/
    inhibit-dryrun.sh    <- the synthetic alert probe

The runbooks/inhibit-review.md is the rubric. The scripts/inhibit-dryrun.sh is the test. The changelog/ directory is the audit trail. The git history is the diff. The four together are the discipline.

A typical review meeting agenda (90 minutes, monthly):

1.  Pull the current alertmanager.yml and the previous
    quarter's changelog.

2.  Run scripts/inhibit-dryrun.sh against the staging
    Alertmanager. Record which rules passed and which
    failed.

3.  For every rule, apply the rubric:
    - Has the rule fired in the last 90 days?
    - Does the source alertname still exist?
    - Does the target alertname still exist?
    - Are the equal labels still present on source and
      target?
    - Has the dependency map changed?
    - Has the label vocabulary changed?
    - Did the dry-run pass?

4.  Classify each finding as dead, inert, stale, drifted,
    over-broad, or retain.

5.  Write the findings to changelog/<current-quarter>.yml.

6.  Open PRs for each action item.

7.  Review the PRs in the next sprint planning.

How to validate it

Three checks. The first is the static count of rules. The second is the dry-run of every rule. The third is the diff against the previous quarter’s changelog.

# 1. Count rules (READ-ONLY). The status API returns the
#    running configuration as a YAML string in
#    .config.original, not as a parsed object.
curl -s http://localhost:9093/api/v2/status \
  | jq -r '.config.original' | sed -n '/^inhibit_rules:/,$p'

Expected output: the inhibit_rules block as YAML. Count the entries and compare against the count from the previous review. A decrease means rules were removed; a change in shape means rules were added or modified.

# 2. Run the dry-run script (READ-ONLY on staging,
#    CONFIGURATION on production if you use synthetic alerts)
./scripts/inhibit-dryrun.sh \
  --alertmanager-url=http://localhost:9093 \
  --config=/etc/alertmanager/alertmanager.yml \
  --output=json

Expected output (illustrative):

[
  { "rule": "HostDown-Suppresses-ServiceDown-Prod",
    "source_fired": true,
    "target_suppressed": true,
    "verdict": "pass" },
  { "rule": "PostgresDown-Suppresses-CheckoutService",
    "source_fired": false,
    "target_suppressed": false,
    "verdict": "fail-source-not-found" },
  { "rule": "RegionalLBDown-Suppresses-Dependents",
    "source_fired": false,
    "target_suppressed": false,
    "verdict": "fail-source-not-found" },
  { "rule": "AuthServiceDown-Suppresses-Dependents",
    "source_fired": true,
    "target_suppressed": true,
    "verdict": "pass" }
]

# 3. Diff against the previous changelog (READ-ONLY)
diff -u changelog/2026-Q3.yml changelog/2026-Q4.yml \
  | head -100

Expected output: a unified diff showing the new findings and the retained rules. The diff is the audit trail’s evidence that the review happened.

How it can fail

Five failure modes specific to the review discipline:

1. The review never happens. Symptom: the changelog directory has entries for Q1 and Q2 but nothing for Q3 or Q4. The rules have not been reviewed in six months. Investigation: ls -la changelog/; if the most recent file is older than 90 days, the review is overdue.
2. The review happens but produces no findings. Symptom: every rule is “retain” for four quarters in a row. The review is a rubber stamp. Investigation: compare the ruleset across quarters; if any rule was added or removed in the last quarter, the previous review should have recorded the change. A pure-retain review is a smell.
3. The dry-run script does not exist. Symptom: the review meeting has no repeatable procedure; each reviewer writes a one-off script. Investigation: confirm scripts/inhibit-dryrun.sh is in the repository and matches the version in the runbook.
4. The change log is not tied to the configuration. Symptom: the changelog says “rule X was removed on 2026-08-15” but alertmanager.yml still contains rule X. The change log drifted from the file. Investigation: every entry in the changelog should correspond to a commit in the git history.
5. The dependency map is not consulted during the review. Symptom: a rule is “retain” because the dry-run passed, but the dependency map changed and the rule is now stale. Investigation: the review runbook must include “diff against the current dependency map” as a step.

How to troubleshoot it

When the review discipline has slipped:

1. Is the changelog current? ls -la changelog/. The most recent file should be within 90 days.
2. Do the rules match the changelog? Pick three rules at random from alertmanager.yml. For each, find the corresponding entry in the most recent changelog. If an entry is missing, the changelog is incomplete.
3. Did the dependency map change? Compare the current service catalog against the labels on every active alert. Any mismatch is a candidate stale rule.
4. Did the dry-run pass? Run inhibit-dryrun.sh and confirm every rule produces the expected suppression.
5. Is the review meeting on the calendar? A team that has slipped the review usually no longer has it on the calendar. The first action is to put it back.

Security implications

The change log is operationally sensitive. It records who changed what, when, and why. A leaked change log is a social engineering target and a map of the team’s operational habits. Treat the changelog directory with the same access controls as the configuration file.

The dry-run script posts synthetic alerts. If the synthetic alert labels are not distinguishable from real alerts, a reviewer reading the API during the dry-run may mistakenly act on the synthetic alert. Prefix every synthetic alert with a label like test="true" or use a dedicated cluster="review" label that no production alert carries.

Performance implications

The review meeting is 90 minutes of human time per month. The dry-run script runs against staging and takes 5 to 15 minutes for a typical ruleset of 20 rules. The change log is a small YAML or Markdown file; the storage cost is trivial.

The cost of not running the review is not visible until an incident. The incident reveals a stale rule that should have been updated; the on-call rota pays the cost in pager fatigue and incident duration.

Production guidance

• Run the review monthly. The cadence is not negotiable. A quarterly review is too slow; a weekly review is too expensive.
• Keep the rubric short. Seven questions, applied mechanically. The meeting should be 90 minutes, not three hours.
• Tie the change log to the git history. Every entry in the change log corresponds to a commit. Every commit has a change log entry. The two together are the audit trail.
• Automate the dry-run. The script is the only repeatable proof that a rule fires. Without the script the review is a discussion.
• Treat the review as a meeting, not an automated job. The value is the human review of the answers to the rubric questions.

Verification

You should now be able to answer:

• What are the seven questions in the inhibit-rule review rubric?
• How do you test that a rule still fires against a live alert set?
• What is the difference between a dead rule, an inert rule, and a stale rule?
• Why must the change log be tied to the git history of the configuration file?

Quiz