Why Inhibition Exists

A production host goes hard-down at 02:14. Within ten seconds the paging channel receives five alerts for one failure: HostDown, NodeExporterDown, ServiceDown, ServiceErrorRateHigh, and LatencyHigh. The on-call engineer wakes to a wall of red. The five alerts share one root cause and one remediation. Four of the five should never have been sent.

Inhibition is the Alertmanager discipline that prevents the wall of red. It declares a relation between two alerts: while alert A is firing, suppress alert B if B matches a set of labels and A matches another set, and the labels they share are equal. Inhibition runs after deduplication and grouping but before routing and notification. It is the last automated chance to cancel an alert whose root cause is already paged.

This lesson frames inhibition as a production discipline. The next five lessons show how to write the rules, validate them, audit them, and stop them from silently masking real failures.

What it is

Inhibition is a rule inside alertmanager.yml that declares one alert as a source and another as a target. While a source alert is active, target alerts whose labels match the rule are marked suppressed and excluded from notification. The source alert itself is unaffected.

Alertmanager pipeline (simplified)

  Prometheus / Mimir
        |
   +----v----+
   | Receive |   <- raw alerts from rule evaluation
   +----+----+
        |
   +----v----+
   |  Group  |   <- by group_by labels
   +----+----+
        |
   +----v----+
   | Inhibit |   <- source matches -> suppress target
   +----+----+
        |
   +----v----+
   |  Route  |   <- matchers decide receiver
   +----+----+
        |
   +----v----+
   | Notify  |   <- silence filter, throttling, webhook
   +----+----+

Inhibition sits between grouping and routing. By the time the router sees the alert set, the consequence alerts have already been suppressed.

Why a sysadmin cares

The pager is a finite resource. Every alert that fires burns on-call attention. A team whose pager fires for the same root cause five times loses:

• Trust in the pager. Engineers start ignoring individual alerts because they assume the cause is already known.
• Signal. The fifth alert adds no information. The first one already pointed at the host.
• Sleep. Five pages at 02:14 is five wake events. One page is one.

Inhibition is not a tuning exercise. It is information design. The set of alerts that survives inhibition should be the smallest set that uniquely identifies the cause and the smallest set the on-call engineer needs to act on.

Two patterns justify inhibition in almost every production environment:

• Upstream-down. A single component (a database, a load balancer, an authentication service) goes down. Every dependent service emits its own failure alert. The dependent alerts are correct, but the operator should only be paged for the upstream.
• Cluster-host. A host dies. Every exporter, every scrape target, every health check on that host fails. The host-down alert is the cause. The scrape-failure alerts are consequence.

Both patterns are addressable by silences. The difference is that silences are time-boxed manual exemptions and inhibition is a standing relation between alerts. Inhibition is the right tool when the relation is structural (host owns services, service depends on database). Silences are the right tool when the relation is one-off (maintenance window, noisy deploy, frozen code).

How it works

An inhibit rule has three label sets:

• source_matchers (or the legacy source_match) — labels that identify the cause alert.
• target_matchers (or the legacy target_match) — labels that identify the consequence alerts.
• equal — a list of label names whose values must be identical in source and target for the inhibition to apply.

The rule fires when at least one alert in the active set matches the source matchers and at least one alert matches the target matchers. For every target match, Alertmanager checks the equal labels. If the values agree, the target is suppressed.

Source alerts  :  alertname=HostDown instance=db-prod-03 severity=critical
Target alerts  :  alertname=ServiceDown instance=db-prod-03 severity=warning
                 alertname=ServiceDown instance=db-prod-04 severity=warning
                 alertname=ServiceErrorRateHigh instance=db-prod-03 severity=warning

equal: [instance]

Result         :  the db-prod-03 ServiceDown and
                  ServiceErrorRateHigh alerts are suppressed.
                  The db-prod-04 ServiceDown is NOT suppressed
                  because its instance does not equal db-prod-03.

The equal clause is what stops an over-broad rule from suppressing unrelated alerts. A rule that says “any HostDown suppresses every warning alert” with no equal: clause would suppress warnings on every host for every host-down event. The equal clause constrains the blast radius.

Under the hood

How to configure it

The minimum viable rule that addresses the cluster-host pattern:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  # When a HostDown alert is firing on instance X, suppress every
  # "Down" or "Error" alert whose instance is X. The equal clause
  # keeps the blast radius on a single host.
  - source_matchers:
      - alertname=~"HostDown|NodeDown|InstanceDown"
      - severity="critical"
    target_matchers:
      - alertname=~".*Down$|.*ErrorRate.*|.*LatencyHigh"
      - severity=~"warning|critical"
    equal: [instance]

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

Three points worth noting in the snippet:

1. source_matchers uses a regex to catch all host-down variants rather than enumerating each alertname. The | alternation pattern is explicit; do not write HostDown alone unless that is the only alertname you own.
2. equal: [instance] is the whole discipline. Without it, the rule suppresses service alerts on every host. With it, the rule suppresses service alerts only on the host that is down.
3. The target alert set excludes severity=info so informational alerts (e.g. a per-host scrape-failure counter) still surface.

How to validate it

Three commands confirm the rule is loaded and behaves as written. Run all three, in order.

# 1. Static syntax check (CONFIGURATION severity)
amtool check-config /etc/alertmanager/alertmanager.yml

Expected output (illustrative):

Checking /etc/alertmanager/alertmanager.yml
  SUCCESS
Parsed and loaded successfully
Found inhibit rule 'HostDown-Suppresses-ServiceDown'
  with 2 source matchers and 2 target matchers

Then reload Alertmanager and confirm via the API:

# 2. SIGHUP reload (CONFIGURATION severity)
kill -HUP "$(pidof alertmanager)"

# 3. Live API check (READ-ONLY)
curl -s http://localhost:9093/api/v2/status | jq -r '.config.original'

The .config.original field echoes back the running configuration as a YAML string, re-marshalled by Alertmanager with secrets replaced by <secret>. Look for the inhibit_rules block and confirm it matches the file on disk.

To prove the rule is firing, post two synthetic alerts to the API with matching instance labels:

# 4. Live behaviour check
amtool alert add alertname=HostDown \
  instance="db-prod-03" severity="critical"
amtool alert add alertname=ServiceDown \
  instance="db-prod-03" severity="warning"

sleep 2
curl -s 'http://localhost:9093/api/v2/alerts?active=true&silenced=true' \
  | jq '.[] | {alertname: .labels.alertname,
                instance: .labels.instance,
                status: .status.state,
                inhibitedBy: .status.inhibitedBy}'

Expected output (illustrative):

{ "alertname": "HostDown",
  "instance": "db-prod-03",
  "status": "active",
  "inhibitedBy": [] }
{ "alertname": "ServiceDown",
  "instance": "db-prod-03",
  "status": "suppressed",
  "inhibitedBy": ["HostDown"] }

If the second alert shows status: "suppressed" with inhibitedBy pointing at the source, the rule is doing its job.

amtool has no command that deletes an alert. Alerts leave Alertmanager by resolving, not by being removed: an alert posted without an explicit end time resolves on its own after the global resolve_timeout (five minutes by default). To resolve one immediately, re-post the same label set with a start and an end that are both in the past.

# Clean up: resolve the two synthetic alerts.
START="$(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ)"
END="$(date -u -d '-1 minute' +%Y-%m-%dT%H:%M:%SZ)"
amtool alert add alertname=HostDown \
  instance="db-prod-03" severity="critical" \
  --start="$START" --end="$END"
amtool alert add alertname=ServiceDown \
  instance="db-prod-03" severity="warning" \
  --start="$START" --end="$END"

# Confirm neither is in the active set any more.
amtool alert query alertname=HostDown

How it can fail

Five failure modes appear repeatedly in production. Each has an observable symptom.

1. No equal: clause; blast radius is the entire fleet. A HostDown on one host silently suppresses ServiceDown on every host that happens to share an alertname. Symptom: curl /api/v2/alerts?active=true shows dozens of inhibitedBy entries that should not be present. Investigation: inspect every suppression and confirm the source and target share the labels you intended.
2. Regex matcher without anchors. alertname=~Service was intended to match ServiceDown but also matches MyServiceHasAProblem and ServiceErrorRateHigh. Symptom: alerts are suppressed on label combinations the rule author never intended. Investigation: grep the alertmanager log for inhibit and check the matched alertname against the pattern.
3. Source alert never fires; rule is dead code. Symptom: the rule appears in amtool check-config output but no suppression has ever occurred. Investigation: query the alert history. A rule that has not fired in 90 days is either unnecessary or misconfigured.
4. Source and target share no equal label; rule is inert. Symptom: the rule is syntactically valid but the target alerts are never suppressed even when the source is firing. Investigation: list both source and target alerts, list the labels they share, confirm those labels are in equal:.
5. Rule suppresses the alert the operator wants to see. Symptom: a critical alert never reaches the pager because a less-specific alert on the same labels is firing first. Investigation: check the source alertname against the target severity; if the source is informational and the target is critical, the rule is inverted and the most important alert is the one being suppressed.

How to troubleshoot it

Work the problem from the alert side backwards to the rule side.

1. Confirm the target is firing. curl /api/v2/alerts?active=true&filter=alertname=ServiceDown. If the target is not in the active set, the problem is not the rule.
2. Confirm the target is suppressed. curl /api/v2/alerts?active=true&silenced=true. The status.inhibitedBy field lists the source alerts that suppressed this target. An empty list means no source alert is matching.
3. Confirm the source is firing. Query the API for the source alertname. The source must be active for the rule to apply. A source that is itself suppressed cannot inhibit anything.
4. Confirm the equal clause is satisfiable. Diff the labels on the source and target. Every label name in equal: must be present on both alerts with the same value.
5. Confirm the matchers are not over-broad. Regex matchers without anchors will surprise you. Write the rule, list the alerts it would match with a small probe (e.g. amtool alert add with a known label set), then test.
6. Read the alertmanager log. Look for lookupSource and matched source lines at debug level (--log.level=debug). The lines show which alert pair triggered each suppression.

If the rule is silently masking alerts and the team is unaware, the production consequence is the worst-case failure mode: a real incident is in progress but only the consequence alert is firing and the consequence is suppressed. The mitigation is the audit trail lesson at the end of this module.

Security implications

The Alertmanager v2 API (/api/v2) exposes the full set of active, suppressed and silenced alerts. Authentication defaults to none when the web config does not include basic_auth, bearer_token, or oauth2. Production deployments must terminate the API behind a reverse proxy with mTLS or basic auth, or expose it only on the loopback interface. An operator who can read /api/v2/alerts can read every suppressed alert and every silenced matcher — including maintenance windows and customer labels.

Inhibit rules themselves contain label patterns. A rule with alertname=~"Internal.*" exposes the naming convention of internal services to anyone with read access to the config repository. Treat alertmanager.yml as configuration that is sensitive to internal naming.

Performance implications

Inhibition is evaluated in memory. Each evaluation is O(sources x targets). A deployment with 500 active alerts and 20 inhibit rules performs roughly 10 000 label comparisons per evaluation, which runs on every alert state change. This is small relative to the cost of routing and notification. The real performance risk is the regex matcher. A regex like alertname=~".*" is a constant-time compile but evaluates against the alertname string on every match. Prefer the equality matcher = when the alertname is known.

Cardinality enters the picture through the equal clause. A rule that includes equal: [instance] evaluates per-instance. With 10 000 hosts this is fine. With 1 000 000 hosts the comparison budget grows linearly with the active set. If your fleet has tens of thousands of hosts, prefer routing-tree grouping to inhibit rules for the bulk of suppression.

Production guidance

• Always validate with amtool check-config before SIGHUP. A bad YAML reload can prevent Alertmanager from starting cleanly.
• Always include equal:. A rule without equal: is rarely correct.
• Always anchor regex matchers. ^HostDown$ is precise; HostDown is not.
• Review rules on a schedule. The last lesson in this module defines the rubric.
• Prefer equality matchers over regex when the alertname is known. Regex is for enumeration across many alertnames.
• Distinguish structural suppression (inhibit) from time-boxed suppression (silence). Do not use a silence as a substitute for an inhibit rule.

Verification

You should now be able to answer:

• What does Alertmanager inhibition do that silences and routing do not?
• When is the upstream-down pattern the right justification for an inhibit rule, and when is a silence the right tool?
• What is the role of the equal: clause, and what happens when it is omitted?
• How do you prove a rule is firing against a live alert set?

Quiz