Inhibition and silences — the alert noise reduction

Inhibition and silences are the alert noise reduction. Inhibition is the conditional suppression; silences are the temporary suppression. This lesson walks the inhibition, the silences, the production patterns, and the failure modes.

The inhibition

The inhibition:

inhibit_rules:
- source_matchers:
  - alertname = ClusterDown
  target_matchers:
  - severity = warning
  equal: [cluster]

The inhibition is the conditional suppression.

The inhibition rules

The inhibition rules:

inhibit_rules:
# When the cluster is down, suppress the warning alerts
- source_matchers:
  - alertname = ClusterDown
  target_matchers:
  - severity = warning
  equal: [cluster]

# When the HPA is at max, suppress the HPA alerts
- source_matchers:
  - alertname = HPAAtMaxReplicas
  target_matchers:
  - alertname = PodScalingFailed
  equal: [cluster, namespace]

# When the node is not ready, suppress the pod alerts on that node
- source_matchers:
  - alertname = NodeNotReady
  target_matchers:
  - alertname = PodNotReady
  equal: [cluster, node]

The inhibition rules are the suppression.

The inhibition matchers

The inhibition matchers:

source_matchers:
- alertname = ClusterDown
- severity = critical

target_matchers:
- severity = warning

The matchers are the conditions.

The equal field

The equal field:

inhibit_rules:
- source_matchers:
  - alertname = ClusterDown
  target_matchers:
  - severity = warning
  equal: [cluster]

The equal field specifies the labels that must match.

The silences

The silences:

amtool silence add --alertmanager http://alertmanager:9093 \
  --comment "Maintenance window" \
  --duration 1h \
  --start "2026-08-16T10:00:00Z" \
  --match "severity=warning"

The silences are the temporary suppression.

The silences via the API

The silences via the API:

curl -X POST http://alertmanager:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false}
    ],
    "startsAt": "2026-08-16T10:00:00Z",
    "endsAt": "2026-08-16T11:00:00Z",
    "createdBy": "ops@example.com",
    "comment": "Maintenance window"
  }'

The silences are created via the API.

The silences expiry

The silences expiry:

# Silence ID from the listing below:
SILENCE_ID=8f1c0d2e-4b6a-4c53-9f7e-2a1b3c4d5e6f

# List the silences
amtool silence list

# Expire a silence
amtool silence expire "$SILENCE_ID"

The silences are managed.

The production patterns

The production patterns:

inhibit_rules:
# Suppress lower-severity alerts when higher-severity fires
- source_matchers:
  - severity = critical
  target_matchers:
  - severity = warning
  equal: [alertname, cluster]

# Suppress node alerts when cluster alert is firing
- source_matchers:
  - alertname = ClusterDown
  target_matchers:
  - severity = warning
  equal: [cluster]

The patterns are the production inputs.

The temporary silences

The temporary silences:

# Silence during a deployment
amtool silence add --alertmanager http://alertmanager:9093 \
  --comment "Deploying v1.34.1" \
  --duration 30m \
  --match "alertname=HighErrorRate,service=nginx"

The temporary silences are the deployment-time suppression.

The failure modes

The common failure modes:

Over-inhibition

The inhibition rules are too strict.
The lower-level alerts are suppressed too often.
The operator misses the underlying issue.

The mitigation is to test the inhibition rules.

Stale silences

The silences are not removed.
The alerts are suppressed for too long.
The operator is unaware of the underlying issue.

The mitigation is to use the temporary silences with expiry.

The cross-course references

• The Alertmanager course covers the routing.
• The Prometheus course covers the alerting.
• The SRE course covers the on-call patterns.

Quiz

Production discipline

• Configure the inhibition rules. The conditional suppression.
• Use the silences. The temporary suppression.
• Avoid over-inhibition. The lower-level alerts.
• Use the temporary silences. The deployment-time.
• Expire the silences. The cleanup.
• Document the configuration. The rules, the silences.

The inhibition and silences are the alert noise reduction. Operating it well is the inhibition rules, the silences, and the production patterns.