Tuning Process

A team has 42 alerting rules. Eighteen of them page on-call. The rest fire into Slack or open tickets. The team tracks false-positive rate in aggregate: 14% across all 42 rules. Three months later the same team looks at it again: 18%. Page-level rules are at 6%, chat rules are at 21%, ticket rules are at 35%. The aggregate hides the imbalance. Page rules are well-tuned. Chat rules are tolerable. Ticket rules are not.

A tuning process that does not separate tiers is a tuning process that cannot see where the work is.

What the tuning process is

The tuning process is the recurring discipline of reviewing, per alert tier, whether the rules at that tier catch real incidents at the rate the team intends and produce false positives at the rate the team can absorb. Tuning is not a single threshold change; it is a steady cadence of measurement and adjust per tier.

There are three tiers in a mature alerting system:

• Page (severity=page). The on-call engineer’s sleep is at stake. The acceptable false-positive rate is single-digit percent. Anything above 5% is a tuning problem.
• Chat (severity=chat, no page). The team sees it in Slack during the working day. The acceptable false-positive rate is higher (10-20%) but the rate still drives dashboard signal.
• Ticket (severity=ticket). A work item is opened in the team’s queue. The acceptable rate is highest (20-35%) but the ticket itself becomes a cost if the noise is high.

The tiers are not a numerical preset. They are a description of how the alert reaches a human and how much human time the alert can absorb without degradation.

Why a sysadmin cares

Tuning is what closes the loop between rule design and production behaviour. Without tuning, the system launches with the rules that were written at the moment of design and never improves. The traffic shape that the rules were calibrated against drifts, and the rules become noise.

A tuning process also makes the false-positive rate visible. The cost of an unknown false-positive rate is the silent trust decay that turns into a missed incident. A team that measures per-tier FP rate every month will see the rate rise a quarter before the team starts to dismiss alerts. A team that does not measure will find out when the dismissals cost them an hour of resolution time on a real outage.

How the process is supposed to work

The tuning process is six activities, in cadence:

        +---------------------+
        | 1. Tag every rule   |
        |    with a tier      |
        +----------+----------+
                   |
                   v
        +---------------------+
        | 2. Configure       |
        |    per-tier routes |
        +----------+----------+
                   |
                   v
        +---------------------+
        | 3. Close every      |
        |    alert with a note|
        +----------+----------+
                   |
                   v
        +---------------------+
        | 4. Measure FP rate  |
        |    per tier monthly|
        +----------+----------+
                   |
                   v
        +---------------------+
        | 5. Tier review      |
        |    meeting (monthly)|
        +----------+----------+
                   |
                   v
        +---------------------+
        | 6. Make tier-aligned |
        |    tune changes      |
        +---------------------+

Step 1. Tag every rule with a tier. The label severity=page|chat|ticket is the standard Prometheus shape. The label is what the route uses to send to the right receiver; it is also what the tier review uses to slice the data.

Step 2. Configure per-tier routes. Each tier has its own repeat_interval (the minimum time between notifications for the same alert), its own grouping, and its own receiver. The first response to a noisy page-tier rule is rarely a threshold change; it is typically a demotion to a chat-tier route.

Step 3. Close every alert with a note (Lesson 05). The note is the data source for the FP-rate measurement. Without notes, the measurement is impossible.

Step 4. Measure FP rate per tier monthly. A rule counts as a true positive if the close-out note contains an incident reference or an explicit “true positive” entry. A rule counts as a false positive if the note contains the word “no action” or “no incident” or a canonical false-positive tag.

Step 5. Tier review monthly. One hour, the team, the per-tier dashboards, the rule changes for the month. Rules above their tier’s FP threshold go onto the review agenda. Promotion between tiers is allowed (a chat rule that has caught no real incidents in six months can move to a slower cadence; a chat rule that catches three real incidents in a month can promote to page).

Step 6. Make the tune changes in the rule file. The change should be tier-aligned: a tier demotion (page → chat) is appropriate when the rule is producing false positives in the page tier; a threshold lift is appropriate when the rule is producing false positives because the threshold is too low for the new normal band; a layer-2 matcher fix is appropriate when the rule is producing false positives because the labels have drifted.

The most common shape

The most common shape is per tier, with a documented threshold above which a rule is escalated to review. Each tier has its own threshold. The numbers are tier-specific. A page rule with a 6% FP rate is high-noise; a ticket rule with a 6% FP rate is acceptable. The same rate reads differently across tiers.

The shape in code:

tier:        page     chat      ticket
FP target:   5%       15%       30%
cadence:     pages    work-day  monthly
                         review

What “per tier” means in practice: a team whose page-tier rate is below 5% but whose ticket-tier rate is above 30% does not have a tuning problem on the page tier; the team has a tuning problem on the ticket tier. The fix is to demote or retire ticket-tier rules that catch no real incidents.

Under the hood

Alertmanager 0.28.x supports per-route configuration including:

• group_by for what to group alerts on
• group_wait for how long to wait before sending the first notification
• group_interval for subsequent notifications in the group
• repeat_interval for the minimum gap between re-notifications for the same alert
• routes for nested routing decisions
• receiver for the named receiver (email, Slack, PagerDuty, webhook)

A per-tier configuration uses a top-level route that matches on severity and dispatches to a tier-specific subroute. Each subroute has its own repeat_interval and its own receiver.

The repeat_interval is the central tunable. A page-tier route might use repeat_interval: 1h so the same alert re-pages at most once an hour; a chat-tier route might use repeat_interval: 4h; a ticket-tier route might use repeat_interval: 24h or no repeat at all (the ticket itself is the persistent representation).

Inhibition is the cross-tier mechanism: a chat-tier alert can be inhibited by a page-tier alert on the same service. This keeps chat-tier chatter low during a real incident.

How to configure it

A canonical Alertmanager configuration with three tiers:

global:
  resolve_timeout: 5m

route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-prod
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 1h
      continue: false
    - matchers:
        - severity="chat"
      receiver: slack-prod-alerts
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      continue: false
    - matchers:
        - severity="ticket"
      receiver: jira-ops
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 60s
      group_interval: 30m
      repeat_interval: 24h
      continue: false

inhibit_rules:
  - source_matchers: [severity="page"]
    target_matchers: [severity="chat"]
    equal: ['alertname', 'cluster', 'service']

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/sink'
  - name: pagerduty-prod
    pagerduty_configs:
      - routing_key: '<PD_KEY>'
  - name: slack-prod-alerts
    slack_configs:
      - api_url: '<SLACK_WEBHOOK>'
        channel: '#prod-alerts'
  - name: jira-ops
    webhook_configs:
      - url: 'http://jira-ops-bridge:8080/'

Six things to read into that:

• Three routes blocks, one per tier. Each has its own repeat_interval. The page tier is fast, the chat tier is half-day, the ticket tier is daily.
• An inhibit_rules block: a page-tier alert on the same alertname, cluster, service inhibits a chat-tier alert. During a real incident, the chat chatter is suppressed.
• The continue: false on each tier route means a matching alert does not also fall through to the next tier. Each alert reaches exactly one receiver.
• group_by is consistent across tiers so that the alerting team’s view of “an alert” matches the operator’s view across tiers.
• The default receiver is intentionally a sink endpoint to catch any alert that does not match a tier — useful during rollouts and as a fallback.

How to validate it

Validate by sending a synthetic firing alert at each tier and confirming the notification reaches the right receiver with the right repeat_interval.

amtool config show --alertmanager.url=http://alertmanager:9093

Sample output should match the routes above.

amtool silence add --alertmanager.url=http://alertmanager:9093 \
  --duration=10m \
  --comment='tier review test' \
  --matcher=alertname=TestPageAlert \
  --matcher=severity=page

amtool silence add --alertmanager.url=http://alertmanager:9093 \
  --duration=10m \
  --comment='tier review test' \
  --matcher=alertname=TestChatAlert \
  --matcher=severity=chat

Inspect the notification history:

curl -s 'http://alertmanager:9093/api/v1/alerts' \
  | jq '.[] | {labels: .labels, status: .status, receivers: .receivers}'

Sample output, confirming tier routing:

{"labels":{"alertname":"TestPageAlert","severity":"page"},"receivers":["pagerduty-prod"]}
{"labels":{"alertname":"TestChatAlert","severity":"chat"},"receivers":["slack-prod-alerts"]}

Track the FP rate per tier using a Prometheus query:

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'sum by (severity) (increase(alertmanager_alerts_received_total[30d]))'

The per-tier rate is then computed by joining this with the team’s close-out note labels.

How it can fail

Six specific shapes, each with the symptom that distinguishes it from other tuning-process failures:

1. No tier labels on rules. Every rule routes to the default receiver; the page-tier rate is unmeasurable. Symptom: Alertmanager UI shows a single receiver for every alert.

2. Single repeat_interval for every tier. The page-tier rule floods at chat-tier rates because the repeat_interval was set once and never reviewed. Symptom: PagerDuty opens new incidents every 4 hours for the same alert even when the alert has been acknowledged.

3. No close-out notes. The FP rate cannot be measured because there is no record of which alerts were real. Symptom: the team discusses the FP rate in anecdotes, not numbers.

4. Tier review deferred repeatedly. The monthly review is on the agenda but never happens; tuning is reactive, when an alert has fired too often. Symptom: tuning only happens after a firefight.

5. Aggregate FP rate used for tier review. The team reads “16% overall” and decides page rules are fine. Symptom: ticket-tier rate is 35%, page-tier rate is 4%; the aggregate hides both.

6. Promotion across tiers without a documented change. A rule was promoted from chat to page because a chat conversation with the on-call engineer felt page-worthy. The rule was not redesigned for the higher cadence. Symptom: the new page-tier rule fires at chat-tier rates and the on-call engineer is over-notified.

How to troubleshoot it

1. Check the severity label is present on every rule. Rules without a severity label are routing to the default receiver.
2. Verify the tier routing matches the team’s intent. Synthetic alerts at each tier should reach the right receiver.
3. Read the close-out notes from the last 30 days. If they are absent, the measurement is impossible and the first thing to fix is the note discipline.
4. Compute the per-tier FP rate. If the page-tier rate is above 5%, the page-tier rules need a focused review; do not let the page-tier rate get averaged away.
5. Run a focused review on the worst tier. Make tier-aligned changes: demote noisy page rules to chat, retire ticket rules with sustained 30%+ rates and no real-incident catches.

Security implications

The receivers in the tier routes are the destinations for alert notifications. Each receiver’s credentials (pagerduty_configs[*].routing_key, slack_configs[*].api_url, webhook_configs[*].url) are sensitive. The Alertmanager configuration file is a secret store; it should be managed as one.

The tier routing should also respect least-privilege. A page-tier route to PagerDuty is acceptable because the on-call engineer needs to know. A ticket-tier route to a public Slack channel is not.

Performance implications

The per-tier routing cost in Alertmanager 0.28.x is small. The grouping decision is per alert and the notification throughput is bounded by the receiver’s capacity. The team’s cost is in the close-out notes and the monthly review.

The Prometheus-side cost of the per-tier labels is negligible: one label per alert. The cost is in the team’s discipline, not the platform.

Production guidance

• Apply the tier label consistently. Rules without a tier are routing defects.
• Run the per-tier FP-rate measurement monthly. Bake it into the team’s review.
• Hold the monthly tier review on a recurring calendar slot. Move it forward when needed; do not cancel it.
• Pair tier tuning with the investigation discipline of Lesson 05.

Verification

You should now be able to answer:

• What are the three tiers of an alerting system and what is the approximate acceptable false-positive rate for each?
• Why is a per-tier false-positive rate more useful than an aggregate rate?
• What is repeat_interval and how does it differ across tiers?
• Why does promotion across tiers need a redesign, not just a label change?

Quiz