Basic Inhibition

The basic inhibition: when HostDown fires, suppress ServiceDown on the same host. The result is one alert (the host) instead of five. The on-call engineer fixes the host; the services recover automatically.

This lesson writes that rule from scratch. It is the most common production inhibit pattern, the one most often copied from a starter config without being read. The lesson walks through the YAML block, the matcher primitives, the most common matcher mistake (regex without anchors), and a repeatable dry-run procedure that proves the rule is doing what its author intended.

What it is

A basic inhibit rule is a single entry under the top-level inhibit_rules: key in alertmanager.yml. The entry declares one alert as the source (the cause), another set of alerts as the target (the consequence), and the labels whose values must be equal for the suppression to apply.

inhibit_rules:
  - source_matchers:
      - alertname="HostDown"
    target_matchers:
      - alertname="ServiceDown"
    equal: [instance]

In this example, while an alert with alertname="HostDown" is active on a given instance, every alert with alertname="ServiceDown" on the same instance is suppressed. Any ServiceDown on a different instance is unaffected.

Why a sysadmin cares

The host-down pattern is the single most common source of alert noise in any platform of meaningful size. A host dies; ten exporters fail; ten scrape-failure alerts fire; the operator is paged ten times for one fix. The pattern repeats on every maintenance, every reboot storm, every cloud-provider incident.

Getting the basic rule right delivers the largest noise reduction per minute of work. Getting it wrong delivers a silent suppression that hides real, unrelated failures. The difference is one matcher syntax decision and one equal: clause.

How it works

The Alertmanager matcher language has four primitives:

  =        exact equality          alertname="HostDown"
  !=       exact inequality        severity!="info"
  =~       regex match             alertname=~"Host.*"
  !~       negative regex match    alertname!~"Test.*"

Equality (=) is the safe default. Use it whenever the matched value is a known, finite set (alertname, severity, cluster).

Regex (=~) is for enumeration across many alertnames. Two rules:

• Anchor the pattern explicitly. ^HostDown$ matches exactly HostDown. HostDown matches HostDown, HostDownForMaintenance, WhyHostDownToday, and SomeOtherHostDownAlert.
• Bound the alternation. ^HostDown|NodeDown$ is a precedence trap — the ^ binds only to the first alternative. The pattern matches “HostDown” or any string ending in “NodeDown”. Wrap each alternative in a group: ^(HostDown|NodeDown)$.

Pattern          :  ^(HostDown|NodeDown)$
Matches          :  "HostDown", "NodeDown"
Does not match   :  "HostDownForMaintenance", "WhyNodeDown"

Pattern          :  ^HostDown|NodeDown$
Matches          :  "HostDown", "WhyNodeDown", "HostDownNodeDown"
Does not match   :  "hostdown", "HostDownForMaintenance"

The matcher is evaluated against each alert’s label values per evaluation. A regex without anchors is a regex that will eventually surprise you.

Under the hood

How to configure it

A real rule for a host owning ten services. Three pieces to note: the regex anchor on the source, the bounded alternation on the target, and the single-label equal: clause.

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  # 1. Host-down alert suppresses per-service down alerts on
  #    the same instance. Anchored regex, equality equal.
  - source_matchers:
      - alertname=~"^(HostDown|NodeDown|InstanceDown)$"
      - severity="critical"
    target_matchers:
      - alertname=~"^(ServiceDown|ExporterDown|ScrapeFailed)$"
      - severity=~"^(warning|critical)$"
    equal: [instance]

  # 2. Service-down alert suppresses service-error alerts on
  #    the same instance. The source here is "critical" so
  #    the rule only suppresses once a real outage is paged.
  - source_matchers:
      - alertname="ServiceDown"
      - severity="critical"
    target_matchers:
      - alertname=~"^(ServiceErrorRateHigh|ServiceLatencyHigh)$"
      - severity="warning"
    equal: [instance, service]

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

Three things to read carefully:

1. Source anchor. ^(HostDown|NodeDown|InstanceDown)$ — the group and the $ anchor ensure the pattern matches only those three alertnames. Removing the anchors would also match MyHostDownAlert and any future alertname that happens to contain “HostDown”.
2. Target severity. The target matchers restrict to warning|critical. The info severity is left alone. An informational alert is allowed to surface even when a service is down, because it might be the early signal of the next problem.
3. equal: [instance, service]. The second rule ties suppression to both instance and service. The service label is what makes the rule specific to a logical service rather than to any service that happens to be down on the same host. Without it, every ServiceErrorRateHigh on the host would be suppressed.

How to validate it

Four checks, in order. The fourth is the live dry-run that proves the rule fires against a real alert.

# 1. Static syntax check (CONFIGURATION severity)
amtool check-config /etc/alertmanager/alertmanager.yml

Expected output (illustrative):

Checking /etc/alertmanager/alertmanager.yml
  SUCCESS
Parsed and loaded successfully
Found inhibit rule 'HostDown-Suppresses-ServiceDown'
Found inhibit rule 'ServiceDown-Suppresses-ServiceErrors'

# 2. SIGHUP reload (CONFIGURATION severity)
kill -HUP "$(pidof alertmanager)"

# 3. Loaded-config echo (READ-ONLY)
curl -s http://localhost:9093/api/v2/status \
  | jq -r '.config.original' | grep -A6 inhibit_rules

The status API returns the running configuration as a YAML string in .config.original, not as a parsed object, so the echo is read as YAML.

Expected output (illustrative):

inhibit_rules:
- source_matchers:
  - alertname=~"^(HostDown|NodeDown|InstanceDown)$"
  - severity="critical"
  target_matchers:
  - alertname=~"^(ServiceDown|ExporterDown|ScrapeFailed)$"
  - severity=~"^(warning|critical)$"
  equal:
  - instance

The fourth check is the live dry-run. Post a synthetic source and a synthetic target with the same instance, then read the API and confirm the target is suppressed.

# 4. Dry-run with synthetic alerts (READ-ONLY on production,
#    CONFIGURATION on staging if amtool is wired to it)
amtool alert add alertname=HostDown \
  instance="db-prod-03" severity="critical"
amtool alert add alertname=ServiceDown \
  instance="db-prod-03" severity="warning"
amtool alert add alertname=ServiceDown \
  instance="db-prod-04" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | select(.labels.alertname=="ServiceDown")
         | {instance: .labels.instance,
            status: .status.state,
            inhibitedBy: .status.inhibitedBy}'

Expected output (illustrative):

{ "instance": "db-prod-03",
  "status": "suppressed",
  "inhibitedBy": ["HostDown"] }
{ "instance": "db-prod-04",
  "status": "active",
  "inhibitedBy": [] }

The db-prod-03 ServiceDown is suppressed; the db-prod-04 ServiceDown is not, because its instance label does not equal the source’s.

amtool has no command that deletes an alert. Alerts leave Alertmanager by resolving, not by being removed: an alert posted without an explicit end time resolves on its own after the global resolve_timeout (five minutes by default). To resolve one immediately, re-post the same label set with a start and an end that are both in the past.

# Clean up: resolve all three synthetic alerts.
START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=HostDown \
  instance="db-prod-03" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  instance="db-prod-03" severity="warning" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  instance="db-prod-04" severity="warning" \
  --start="$START" --end="$END"

amtool alert query alertname=ServiceDown

How it can fail

Five failure modes specific to the basic rule:

1. Unanchored regex on the source matcher. alertname=~Service was intended to match ServiceDown but actually matches ServiceDown, ServiceErrorRateHigh, MyServiceHasAProblem, and NoServiceFound. Symptom: alerts with names the author did not intend are being suppressed. Investigation: dump /api/v2/alerts?silenced=true and look at the matched alertnames.
2. equal: label is not present on the source. The rule declares equal: [region], but the source alert has no region label (the Prometheus rule forgot to add it). Symptom: the target is never suppressed. Investigation: curl /api/v2/alerts?active=true for the source and confirm region is present in .labels.
3. Precedence trap in alternation. alertname=~"^HostDown|NodeDown$" was intended to match either, but actually matches any string starting with “HostDown” or any string ending in “NodeDown”. Symptom: unexpected suppressions. The fix is grouping: ^(HostDown|NodeDown)$.
4. Source alert resolves before the dry-run completes. The synthetic source alert has for: 5m and the test runs at minute zero. Symptom: no suppression appears; the rule is correct but the test is faulty. Wait the full for: duration, or post the alert with no for: for the dry-run only.
5. YAML reload fails silently. A trailing space or a tab character inside a matcher string causes the file to parse but the rule to be ignored. Symptom: amtool check-config succeeds but the rule does not fire. Investigation: compare the file on disk against the .config.original echo from /api/v2/status. A mismatch means the reload did not actually load.

How to troubleshoot it

The diagnostic order is mechanical. Do not skip steps.

1. Is the rule in the loaded config? curl /api/v2/status | jq -r '.config.original' and read the inhibit_rules block out of the YAML it prints. If the rule is missing, SIGHUP did not pick it up. Check the alertmanager log for parse errors.
2. Is the source firing? Query /api/v2/alerts?active=true for the source alertname. If the source is not active, no inhibition will fire. The rule is correct; the source is missing.
3. Is the target firing AND suppressed? Query /api/v2/alerts?silenced=true. The .status.inhibitedBy field on the target tells you which source suppressed it. An empty list means the rule did not fire for that target.
4. Are the matchers matching as you expect? Reproduce in isolation. Pick a known alert, list its labels, and walk through the matcher by hand. The regex matcher in particular: use a one-liner like echo "HostDownForMaintenance" | grep -E '^HostDown$' to prove the pattern.
5. Is the equal: clause satisfiable? For every label in equal:, the value on the source and target must be byte-identical. A trailing space or a capitalisation difference (prod vs Prod) is enough to break the rule.
6. Did the SIGHUP actually load the file? The .config.original field echoes back the running configuration as Alertmanager re-marshalled it, with secrets replaced by <secret>, so compare the inhibit_rules block rather than the whole file. A reload that succeeded but loaded an older file is a classic cause of “the rule works in staging, not in production.”

Security implications

The Alertmanager v2 API is unauthenticated by default. An operator with read access to /api/v2/alerts can enumerate every suppressed alert and read the label values — including customer labels and internal naming conventions. Lock the API behind a reverse proxy with mTLS or basic auth, or bind it to the loopback interface.

The configuration file itself is sensitive. A rule with alertname=~"Internal.*" exposes the internal service naming convention. A rule with severity=~"customer-.*" exposes the severity vocabulary used for customer-impact alerts. Treat the file as configuration that is sensitive to internal naming and restrict repository access accordingly.

Performance implications

Basic inhibition is cheap. A rule with two source matchers, two target matchers, and one equal label is roughly four string comparisons per active alert per evaluation. The cost grows linearly with the active alert set, which is bounded by the rate of state changes — not by the total number of alerts that have ever fired. For a fleet of 10 000 hosts producing 200 alerts per minute, the inhibitor spends a few milliseconds per evaluation and well under 100 MiB of resident memory.

Regex matchers cost more than equality matchers. A regex like ^(HostDown|NodeDown|InstanceDown)$ compiles to a finite-state machine with a few dozen states; the per-match cost is the walk through that machine. Avoid regex matchers whose pattern includes .* at the start of the alternation — the machine backtracks and the per-match cost rises.

Production guidance

• Anchor every regex. Every time. No exceptions.
• Test every rule change with the synthetic-alert dry-run. The cost is 30 seconds. The cost of an unverified rule is hours of incident.
• Diff the loaded config against the file on disk after every SIGHUP. A reload that succeeded but loaded a different file is a real production failure mode.
• Review rules when the dependency map changes. The last lesson in this module defines the rubric.

Verification

You should now be able to answer:

• What is the role of source_matchers, target_matchers, and equal: in a basic inhibit rule?
• Why is ^HostDown$ different from HostDown in production terms?
• How do you prove the rule is firing against a live alert set?
• What is the precedence trap in ^A|B$ alternation, and how do you avoid it?

Quiz