Inhibition

A whole database cluster goes dark. ClusterDown fires. Immediately after, every host-level rule fires: HostDown on each of the eight database hosts, PostgresReplicationLag on each replica, DatabaseConnectionPoolExhausted on the primary, BackupFailed on the backup job. Eight hosts, four alert names, thirty-two firing alerts. One root cause.

Inhibition is the rule that says: when ClusterDown is firing, none of the host-level alerts get a notification. The ClusterDown page tells the on-call the right story. The host-level alerts are still in nflog for the postmortem — they are just suppressed as notifications.

What it is

Inhibition is the mechanism by which Alertmanager silences one set of alerts when another set is firing. It is configured under the top-level inhibit_rules: key and applies globally across all routes. Each rule has:

• source_matchers: (or source_match:) — the matchers that identify the cause alert. When an alert matching these is firing, it can suppress other alerts.
• target_matchers: (or target_match:) — the matchers that identify the consequence alerts that should be silenced while the source is firing.
• equal: (optional) — a list of label names that must have the same value on both the source and the target for the inhibition to apply. The single most important lever for avoiding accidental scope.

Inhibition runs after grouping, before notification. The target alert is still present in nflog/ and is still visible in the UI; it is simply excluded from notifications while a matching source alert is firing.

Why a sysadmin cares

Inhibition is the highest-leverage / highest-risk knob in the whole alertmanager.yml. The leverage is real: a well-designed inhibit rule converts a thirty-two-page flood into a single focused page. The risk is also real: an inhibition rule with a missing or wrong equal: clause can swallow a real incident.

Two failure shapes appear repeatedly:

1. The ClusterDown page hides a real, independent HostDown. A switch reboots. The whole cluster goes dark. The ClusterDown page is right and useful. Meanwhile, an unrelated host on the same cluster has a hardware failure that would normally page a different on-call. The HostDown alert is inhibited by the cluster-level alert. The hardware failure goes unnoticed until the next morning’s health check.
2. The Critical-inhibits-Warning rule swallows a unique critical. A severity=critical source matcher inhibits all severity=warning targets. The intended behaviour is “if the critical page is already out, don’t spam warnings.” The unintended behaviour is that any critical, on any host, silences warnings across the fleet. A critical on the edge router silences warnings on the database cluster.

Both shapes are the same mistake: missing or wrong equal: constraints. The discipline is to write the equal: clause before writing the matcher clause, and to treat any inhibit_rule without equal: as a code review blocker.

How it works

  alert X: alertname=ClusterDown cluster=prod-eu-1 severity=critical (firing)
  alert Y: alertname=HostDown cluster=prod-eu-1 severity=critical (firing)
  alert Z: alertname=DiskFillingSoon cluster=prod-eu-1 severity=warning (firing)

  inhibit_rule:
    source_matchers: [alertname=ClusterDown]
    target_matchers: [severity=critical]
    equal: [cluster]

  evaluation for Z:
    source matches ClusterDown (alert X)?
      yes, X is firing with cluster=prod-eu-1
    target matches severity=critical?
      Z is severity=warning, NO MATCH
    Z is NOT inhibited.

  evaluation for Y:
    source matches ClusterDown (alert X)?
      yes, X is firing with cluster=prod-eu-1
    target matches severity=critical?
      Y is severity=critical, MATCH
    equal: cluster=prod-eu-1 == cluster=prod-eu-1? YES
    Y IS inhibited.

The rule above inhibits critical host-level alerts while the cluster is down — but only inside the same cluster. A critical on the edge router does not inhibit a warning on the database cluster because the cluster label differs.

The equal: clause is the boundary. Without it, the rule matches across the whole fleet.

Under the hood

Inhibition runs as a pass over the active alerts in nflog. For each (source, target) pair, AM checks:

1. Is the source alert firing? (A pending source alert does not inhibit.)
2. Do the source matchers match the source alert?
3. Do the target matchers match the target alert?
4. For every label in equal:, is the value the same on both?

The cost is O(sources x targets) per inhibit_rule per notification cycle. With a handful of rules and a few hundred firing alerts, the cost is negligible.

A subtle detail: a source alert is only a source if it is firing — not pending, not inactive. An alert that has been pending for 30 seconds and has not yet entered firing does not inhibit anything. This matters when you tune for: in the rule definitions: a long for: on the source delays the inhibition window.

How to configure it

A production-grade set of inhibit rules, scoped by equal::

inhibit_rules:
  # Critical pages silence warnings inside the same cluster.
  # Use this to prevent the "one root cause, thirty pages" shape.
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity =~ "warning|info"
    equal: ['cluster', 'alertname']

  # ClusterDown silences host-level alerts inside the same cluster.
  - source_matchers:
      - alertname = ClusterDown
    target_matchers:
      - alertname =~ "HostDown|ServiceDown|InstanceUnreachable"
    equal: ['cluster']

  # DatabaseClusterDown silences database host alerts inside the
  # same database cluster.
  - source_matchers:
      - alertname = DatabaseClusterDown
    target_matchers:
      - alertname =~ "PostgresReplicationLag|PostgresConnectionsExhausted|BackupFailed"
    equal: ['dbcluster']

  # NetworkPartitionDown silences application-level latency alerts
  # that ride on the same network path.
  - source_matchers:
      - alertname = NetworkPartitionDown
    target_matchers:
      - alertname =~ "HighRequestLatency|HighErrorRate"
    equal: ['region']

Two patterns to read off these rules:

• equal: always carries at least one label. The first rule uses cluster and alertname so that a critical on one alert does not silence warnings on a different alert in the same cluster.
• Each rule is a parent/child relationship with a scope. The rule does not say “inhibit this name.” It says “when this parent fires, silence this child inside this scope.” The scope is the equal: label.

The right discipline is to enumerate the alerts, identify the parent/child relationships, and write a rule per relationship with the equal: clause that bounds the scope to where the parent and child are causally related.

How to validate it

amtool config routes test does not directly simulate inhibition. The right approach is to:

1. Fire a real source alert (or simulate with amtool alert add).
2. Fire a real target alert (or simulate).
3. Confirm the target is suppressed via the AM log and via GET /api/v2/alerts.

# 1. Fire the source alert.
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=ClusterDown cluster=prod-eu-1 severity=critical
# Output: component=active stage=active alerts=[ClusterDown]

# 2. Fire the target alert.
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=HostDown cluster=prod-eu-1 instance=db-1 severity=critical
# Output: component=active stage=active alerts=[HostDown, ClusterDown]

# 3. Inspect via the API. The target is still in nflog but
#    its status is "suppressed" while the source is firing.
curl -s http://localhost:9093/api/v2/alerts \
  | jq '.[] | {labels: .labels, status: .status.state}'
# Output:
# {
#   "labels": {"alertname": "ClusterDown", "cluster": "prod-eu-1", ...},
#   "status": "active"
# }
# {
#   "labels": {"alertname": "HostDown", "cluster": "prod-eu-1", ...},
#   "status": "suppressed"
# }

# 4. Resolve the source. The target should become active.
amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=ClusterDown cluster=prod-eu-1 severity=critical \
  --end

The visible state is the truth. status.state == "suppressed" is the unambiguous signal that inhibition is working. If the target stays active, the rule is misconfigured — most likely the equal: clause is wrong or the matchers do not match the alert labels.

How it can fail

The recurring failure modes, in descending order of operational cost:

1. Missing equal: clause. A rule of the form source: severity=critical; target: severity=warning with no equal: silences every warning on the fleet whenever any critical fires. A critical on the cache layer silences the disk-full warning on the database. The fix is to require equal: [cluster, alertname] (or similar) on every rule.
2. equal: carries the wrong label. A rule uses equal: [region] when the parent/child relationship is scoped by cluster. A critical in cluster=prod-eu-1 inhibits warnings across the whole region=eu-west region. Always pick the smallest scope that captures the causal relationship.
3. Source alert is pending, not firing. A long for: on the source rule delays when the inhibition kicks in. In the interim, the targets fire and notify. Tune for: on the source down to a few seconds for the relationship to feel immediate.
4. The source rule never fires. A typo in the source matcher (alertname = ClusterDwon) means the rule never matches. Targets fire and notify normally. The intended inhibition never happens. Validate with amtool config routes test and confirm against the live API.
5. Source alert suppressed by another rule. Inhibition rules do not chain through other inhibition rules in a way that re-suppresses targets. If ClusterDown is itself suppressed by a higher-level rule, the host-level alerts are not inhibited. Verify the firing state of the source via the api/v2/alerts endpoint, not via the visible notifications.
6. Inhibit rule scope matches a sibling, not a child. A parent alert that is firing on a sibling scope (different cluster) inhibits a target that happens to share the severity label. The fix is to add the right equal: label. Always validate by firing two alerts on different scopes and confirming the target is not suppressed.

How to troubleshoot it

When a real alert is missing because of inhibition, the diagnostic order:

1. Confirm the alert is in nflog. curl -s http://localhost:9093/api/v2/alerts | jq '.[] | select(.labels.alertname == "HostDown")'. If it is not present, the rule never fired in Prometheus.
2. Inspect status.state. A state of suppressed confirms inhibition is the cause. A state of active means the notification path is broken elsewhere.
3. Identify the source alert. Filter nflog for severity = critical (or whatever the source matchers are). If no source is firing, the target should not be inhibited.
4. Walk the inhibit rule by hand. For each inhibit_rule, check the four conditions: source firing, source matchers, target matchers, equal labels.
5. Tail the AM log. The line component=inhibitor ... shows the evaluation. Cross-check against the expected behaviour.

Security implications

Inhibition is not security-sensitive on its own, but two adjacent risks matter:

• Suppression as a denial-of-service surface. An attacker who can fire an arbitrary alert (via a Prometheus instance with a permissive alert rule, or via a misconfigured webhook receiver that accepts alerts) can suppress legitimate notifications by firing the source matcher. The control is the same as for any alert source: only trusted sources can push alerts into the pipeline.
• Information disclosure in inhibited alerts. An inhibited alert is still in nflog and visible to anyone with API access. A high-severity security alert that is inhibited by a routine cluster-down alert is still visible to operators with the right permissions; the concern is that non-operators may also see it via the receiver if the inhibition rule is wrong.

Performance implications

Inhibition is a single linear pass over the active alerts per rule per notification cycle. The cost is negligible for fleets of a few thousand firing alerts. The failure shape is a rule with extremely broad source matchers (e.g. severity = .*) that matches every alert — the cost becomes quadratic in the number of firing alerts. Treat any matcher with =~ covering many values as a smell.

Production guidance

• Every inhibit_rule carries an equal: clause. Make it a lint rule in CI: amtool check-config plus a wrapper that parses the YAML and asserts the equal: key is present.
• Prefer narrow source matchers. The source should be a specific alert name, not a severity. alertname = ClusterDown is better than severity = critical.
• Validate with synthetic alerts. A small library of amtool alert add commands that exercise every inhibit rule should run in CI after every config change. The expected status of each target alert should be suppressed in the assertions.
• Document the parent/child relationships. A diagram of the inhibition tree in the instrumentation guide is the cheapest way to keep the relationships straight when the rules grow.
• Audit quarterly. Enumerate all inhibit_rules and confirm each source matcher still corresponds to a real, firing alert. Decommissioned alert names that remain in inhibit rules are silent no-ops; missing names are silent failures.

Verification

You should now be able to answer:

• What are the four conditions that must all be true for inhibition to apply?
• Why is a missing equal: clause a production failure mode?
• What is the difference between a firing source and a pending source in inhibition?
• How do you confirm via the API that an alert is being suppressed by inhibition rather than missing?
• What is the right discipline for designing the equal: clause on a new inhibit rule?

Quiz