Silences and Mutes

A planned database migration will take the primary offline for two hours. Three alerts will fire: PostgresConnectionsExhausted, PostgresReplicationLag, BackupFailed. Without a silence, each fires every group_interval for the full two hours, producing hundreds of notifications. The on-call team is not the team that scheduled the migration. The team that scheduled the migration is the team that needs the silence.

A silence is the right tool: match the cluster and the alert names, set the window to cover the migration, expire it automatically at the end. Two hours later, the silence expires, the alerts start firing again, and a real outage finds an active pipeline.

What it is

A silence is a time-bounded mute of alerts that match a set of label matchers. It is stored in the silences/ directory on disk and indexed in memory. Each silence has:

• A set of matchers (key, op, value) that define which alerts the silence applies to. Ops are =, !=, =~, !~.
• A startsAt timestamp. Defaults to creation time.
• An endsAt timestamp. Mandatory. The silence stops applying at this time, regardless of whether the alert is still firing.
• A createdBy string. Recorded for audit.
• A comment string. Mandatory in the UI; recommended for every silence. Should explain why the silence exists and what to do when it expires.

A silence does not delete alerts. It does not suppress them in nflog. It only prevents them from generating notifications while it is active and matches.

Why a sysadmin cares

Silences are how planned maintenance and known-issue suppressions are expressed in the alerting pipeline. The failure modes are not loud. They are quiet.

1. The silence that swallowed the real incident. A silence is created with matchers: [\{name: "alertname", value: "HostDown", isRegex: false\}] and no other constraints. A planned reboot of a single host covers a hundred unrelated HostDown alerts across the fleet. The real HostDown on the decommissioned host three months later is silenced for the remaining duration of the original maintenance window.
2. The silence without an end. A UI bug, an inattentive operator, or a deliberate “I’ll add the end time later” decision creates a silence with a very long end time. The alert is muted for a year. The team forgets. The alert stops being useful.
3. The forgotten audit trail. Silences are mutable. The audit trail (who created, who edited, who expired) lives in the comment field. A silence with a comment of "ok" is operationally invisible six months later.

The discipline is to write a silence as if you are handing it to yourself at 03:00 with no context: matchers scoped, end time set, comment explaining the why.

How it works

  alert X: alertname=PostgresConnectionsExhausted cluster=prod-eu-1
           severity=critical (firing)

  silence S:
    matchers: [{name: alertname, value: PostgresConnectionsExhausted},
               {name: cluster,   value: prod-eu-1}]
    startsAt: 2026-08-14T02:00:00Z
    endsAt:   2026-08-14T04:00:00Z
    createdBy: dba-team@example.com
    comment: "Planned Postgres major-version upgrade on prod-eu-1 primary"

  evaluation at 03:00:
    alert X matches S.matchers? YES
    current time is between startsAt and endsAt? YES
    X is muted.

  evaluation at 05:00:
    current time is past endsAt? YES
    X is NOT muted. Notification resumes.

A silence has no effect on nflog. The alert is still considered firing, still appears in GET /api/v2/alerts, still shows in the UI as active. It is simply excluded from notifications while the silence is active and matches.

Under the hood

Silences are stored under <storage.path>/silences/. Each file is a binary-encoded silence record. The in-memory index is a flat list scanned per alert on every notification cycle. With hundreds of active silences and thousands of firing alerts, the cost is a few hundred microseconds per notification cycle — negligible.

A silence that has expired (current time past endsAt) is retained on disk and visible via GET /api/v2/silences?silenced=false. Expired silences do not affect alerts. The cleanup of expired silences from disk is governed by --data.retention (default 120h); older silence files are garbage-collected.

A silence can be created without a startsAt (defaults to “now”), without createdBy (defaults to empty), and without a comment (defaults to empty via API; the UI requires it). The only mandatory field is endsAt. The API will reject a silence without endsAt. The UI will reject it visibly.

A silence can be expired before its endsAt via DELETE /api/v2/silence/{id}. The record stays on disk as expired; it is removed from the in-memory index.

How to configure it

Silences are not configured in alertmanager.yml. They are runtime state, created via the UI, the API, or amtool.

The three operational patterns:

1. Planned maintenance silence. Created during the change window. Matchers scoped to the affected cluster and alert names. End time matches the planned recovery. Comment names the change ticket and the owner.
2. Known-issue silence. Created when a non-critical issue is being investigated. Matchers scoped to the specific alert and instance. End time matches the agreed investigation deadline. Comment names the issue tracker.
3. Migration silence. Created during a long migration. Matchers scoped to the migration target. End time matches the migration window. Comment names the runbook.

The wrong patterns (avoid):

• Silence as a substitute for fixing the alert. “The alert is noisy, let us silence it for a year.” This hides the problem. Fix the alert or remove it.
• Silence as a substitute for fixing the rule. “The threshold is wrong, let us silence the alert.” Adjust the threshold instead.
• Silence with no comment. “Test.” “Ok.” “Silence.” These are operationally invisible at audit time.

How to validate it

The validation flow has three steps: list, query, and confirm.

# 1. List all silences (active and expired).
#    Severity: READ-ONLY.
curl -s http://localhost:9093/api/v2/silences | jq '.[].id'
# Output:
# "abc123-def4-..."
# "deadbeef-0001-..."

# 2. Filter to only active silences.
curl -s 'http://localhost:9093/api/v2/silences?silenced=true' \
  | jq '.[] | {id: .id, matchers: .matchers, endsAt: .endsAt}'
# Output:
# {
#   "id": "abc123-def4-...",
#   "matchers": [
#     {"name": "alertname", "value": "PostgresConnectionsExhausted", "isRegex": false, "isEqual": true},
#     {"name": "cluster", "value": "prod-eu-1", "isRegex": false, "isEqual": true}
#   ],
#   "endsAt": "2026-08-14T04:00:00Z"
# }

# 3. Confirm a specific alert is silenced.
curl -s http://localhost:9093/api/v2/alerts \
  | jq '.[] | select(.labels.alertname == "PostgresConnectionsExhausted")
        | {labels: .labels, status: .status.state}'
# Output:
# {
#   "labels": { "alertname": "PostgresConnectionsExhausted", ... },
#   "status": "suppressed"
# }

status.state == "suppressed" confirms the alert is in nflog and not being notified. The distinction between silenced and inhibited is visible only by querying the silences list and the inhibit_rules.

amtool silence add is the CLI workhorse:

# Create a silence via the CLI.
# Severity: CONFIGURATION (writes a silence to the running AM).
amtool --alertmanager.url=http://localhost:9093 silence add \
  alertname=PostgresConnectionsExhausted cluster=prod-eu-1 \
  --start "2026-08-14T02:00:00Z" \
  --end "2026-08-14T04:00:00Z" \
  --comment "Planned Postgres upgrade on prod-eu-1 primary (CHANGE-1234)"
# Output:
# silence_id=abc123-def4-...

# Expire it early if the maintenance finishes ahead of schedule.
amtool --alertmanager.url=http://localhost:9093 silence expire abc123-def4-...
# Output:
# silence expired

How it can fail

The recurring failure modes, in descending order of operational cost:

1. Over-broad matchers. A silence with alertname=HostDown and no other matcher mutes every HostDown alert across the fleet. The right discipline is to require at least two matchers (the alert name and a scope label like cluster or instance) for every silence.
2. A silence without an endsAt. The API requires it; the UI may allow it via direct form input. The silence mutes the alert forever, until someone notices and expires it. Always set endsAt explicitly.
3. A silence whose endsAt is in the past at creation time. The UI may create a silence that has already expired. It has no effect. The operator assumes the silence is active and stops investigating the alert. Verify with GET /api/v2/silences?silenced=true after creation.
4. An expired silence that has not been cleaned up. A silence with endsAt two years ago is still on disk. It is not matching, but it is visible in the UI and confuses the audit review. Audit and clean expired silences quarterly.
5. A silence that mutes a real alert during a parallel incident. A planned maintenance silence on cluster=prod-eu-1 mutes all severity=critical alerts in the cluster. A genuine unrelated critical on a different service in the same cluster is also muted. The discipline is to scope silences tightly: include the specific alert name in the matchers, not just the cluster.
6. A silence created by an ex-employee whose credentials still work. The audit trail (createdBy) records the email address. Six months later the team cannot contact the creator to ask “why is this still here?” The discipline is to require service-account identities for long-lived silences and to expire credentials of departed employees promptly.

How to troubleshoot it

When an alert is missing from notifications and silence is the suspected cause:

1. Query the active silences. curl -s http://localhost:9093/api/v2/silences?silenced=true.
2. For each silence, check whether the alert’s labels match the matchers. A quick jq filter:

   curl -s http://localhost:9093/api/v2/silences?silenced=true \
     | jq '.[] | select(.matchers[]
           | select(.name == "alertname" and .value == "HostDown"))'

3. Inspect the silence’s endsAt. If it is in the future, the silence is the cause. If it is in the past, the silence has expired and is not the cause — look elsewhere.
4. Inspect the alert’s status.state. A state of suppressed confirms the alert is in nflog but not being notified. The cause is either a silence or an inhibit_rule. Distinguish by querying the silences list and the AM log line for inhibition.
5. If the silence must be expired early: amtool silence expire <id> or DELETE /api/v2/silence/<id> via the API.

Security implications

• The silence API is write-capable. A user with access to POST /api/v2/silences can mute any alert. The control is RBAC at the reverse-proxy level (basic auth, OAuth, mTLS) and audit logging of the API.
• The audit trail is the comment field. It is mutable. A bad actor can edit the comment of a long-lived silence to obscure its origin. The control is to log the API calls themselves (nginx access logs, ingress controller logs) so the change is auditable even if the comment is not.
• Silences are state, not configuration. They do not appear in source control. The control is a weekly export of GET /api/v2/silences to a versioned store (S3 with versioning, Git as JSON files) so a forensic question six months later can be answered.

Performance implications

Silences are evaluated per alert per notification cycle. The cost is linear in the number of active silences. Hundreds of silences with thousands of firing alerts produce no measurable overhead. Tens of thousands of active silences start to show up in CPU profiles — at that point, the silences list is itself the operational smell and should be pruned.

The disk cost of the silences directory is one small file per silence (a few hundred bytes). The cleanup window is --data.retention (default 120h).

Production guidance

• Require at least two matchers per silence. Make this a review-time rule. The two matchers should be the alert name and a scope label (cluster, instance, team).
• Always set endsAt. Refuse any silence request that does not have a clear end time. The API already requires it; the UI workflow should make it impossible to omit.
• Comment discipline. Every silence comment should name the change ticket, the owner, and the recovery plan. Treat the comment as a runbook entry.
• Audit weekly. A short script that lists all active silences, groups them by createdBy, and flags any without a comment or with an endsAt more than 30 days out.
• Export silences daily. A snapshot of GET /api/v2/silences to a versioned store gives you a forensic record without depending on the AM disk.

Verification

You should now be able to answer:

• What fields are mandatory on a silence, and which are recommended?
• How do you create a silence via the API, and how do you expire it early?
• What is the difference between an alert being inhibited and an alert being silenced?
• Why is a silence without a scope label a production failure mode?
• What audit trail exists for a silence, and how do you export it?

Quiz