The Page vs Ticket Decision

A page wakes a human. The human is pulled out of whatever they were doing - sleep, dinner, a child’s birthday - and asked to context-switch into an incident. The cost is not the ten minutes of investigation; it is the twenty minutes of waking, the hour of re-focusing the next morning, and the second-order cost of an engineer who has learned to dread their rotation. Tickets do not have this cost. Tickets wait politely in a backlog until the working-hours team picks them up.

The page-vs-ticket decision is therefore the central cost lever in the alerting system. Get it right and the on-call rotation is useful. Get it wrong and the rotation is burned out within a quarter, or - worse - the team has stopped responding to pages entirely.

What it is

A page is a synchronous notification that demands a human response within minutes. The notification is delivered through a channel the on-call engineer cannot ignore (phone call, SMS, push notification with critical severity, PagerDuty high-urgency). The response expectation is “look at this now.” A ticket is an asynchronous work item that joins the working-hours backlog. The notification is delivered through email, Jira, Linear, or a Slack channel the team reads during the day. The response expectation is “address this week.”

The two are not the same thing with a different label. They are two different operational categories with two different cost profiles and two different response expectations.

Why a sysadmin cares

A misrouted alert is one of two errors, both expensive. A false-positive page wakes the on-call for nothing; a ticket that should have been a page lets a real incident sit in Jira for nine hours. The first error accumulates over weeks; the second is a single event that consumes the entire error budget for the month. The decision matrix below is the discipline that prevents both.

How it works

Three axes decide page or ticket. The matrix is short enough to keep on a single index card.

  Page or ticket?
  ================

  Axis 1: User-impact volume
    - More than 5% of users affected       -> page weight +2
    - 1% to 5% affected                    -> page weight +1
    - Internal-only                       -> page weight +0

  Axis 2: Urgency (time-to-mitigate budget)
    - Mitigation window under 15 minutes  -> page weight +2
    - 15 minutes to 4 hours                -> page weight +1
    - More than 4 hours                   -> page weight +0

  Axis 3: Business criticality
    - Revenue / contractual SLA path      -> page weight +2
    - Internal productivity path          -> page weight +1
    - Background / batch                  -> page weight +0

  Score 4 to 6  -> PAGE
  Score 2 to 3  -> TICKET
  Score 0 to 1  -> DASHBOARD PANEL (or delete)

A real example: checkout 5xx rate above 2% affects every user attempting a purchase (volume +2), requires mitigation within the SLO window or the error budget burns (urgency +2), and is on the revenue path (criticality +2). Score 6 - page. The CPU on the batch analytics host above 90% affects no users (volume +0), can wait until the morning standup (urgency +0), and is on the productivity path (criticality +1). Score 1 - dashboard panel.

How to configure it

The Alertmanager route tree that implements the decision:

# alertmanager.yml (excerpt)
route:
  receiver: default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Pages: short repeat, narrow grouping, dedicated receiver
    - matchers:
        - severity = "page"
      receiver: pagerduty
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      continue: false
    # Tickets: long repeat, looser grouping, working-hours receiver
    - matchers:
        - severity = "ticket"
      receiver: jira
      group_wait: 5m
      group_interval: 30m
      repeat_interval: 4h
      continue: false
    # Dashboard candidates: no receiver, just appears in UI
    - matchers:
        - severity = "info"
      receiver: blackhole
      group_wait: 10m
      group_interval: 1h
      repeat_interval: 24h
receivers:
  - name: pagerduty
    pagerduty_configs:
      - service_key: '<redacted>'
        severity: 'critical'
  - name: jira
    webhook_configs:
      - url: 'https://jira.example.com/webhooks/observability'
  - name: blackhole
    # No integrations; alerts visible only in Alertmanager UI.
  - name: default
    slack_configs:
      - api_url: '<redacted>'
        channel: '#observability-noise'

The three routes map cleanly to the decision matrix. severity: page goes to PagerDuty with a one-hour repeat. severity: ticket goes to Jira with a four-hour repeat. severity: info goes to a blackhole receiver with no integration - the alerts appear in the Alertmanager UI for the daily review but do not notify anyone.

The Prometheus rule file is unchanged from the prior lesson. The difference is in the label and the route.

How to validate it

Test the route tree against synthetic alert label sets:

# SEVERITY: READ-ONLY
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname="CheckoutErrorBudgetBurn" \
  severity="page" team="payments" service="checkout"

Expected output ends with -> pagerduty.

# SEVERITY: READ-ONLY
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname="CheckoutHostCPUHigh" \
  severity="ticket" team="payments" service="checkout"

Expected output ends with -> jira.

Validate the Alertmanager config syntax:

# SEVERITY: READ-ONLY
amtool check-config /etc/alertmanager/alertmanager.yml

Reload Alertmanager without a process restart:

# SEVERITY: SERVICE-IMPACT (Alertmanager reload re-routes
# in-flight alerts; brief delivery delay possible)
curl -X POST http://alertmanager:9093/-/reload

Inspect the in-flight alerts and their assigned receivers in the Alertmanager UI under Status -> Active Alerts; the Receiver column must match the route decision.

How it can fail

Four failure modes recur when teams get the decision wrong:

1. Page on cause. A host CPU alert pages because the host crosses 85%. The CPU spike resolves itself in twenty minutes; the on-call loses sleep for nothing. Symptom: high ack rate with low action rate; the on-call learns that pages are optional.
2. Ticket on symptom. An HTTP 5xx alert tickets because the threshold is “above 2%”. The real incident waits nine hours in Jira. Symptom: error budget exhausted before anyone looks; the SLO breach is the first signal the team sees.
3. No receiver differentiation. All alerts route to Slack #oncall. The on-call learns to ignore the channel because half the messages are tickets. Symptom: Slack scrollback is ignored; pages get lost in the noise.
4. Repeat interval mis-set. Tickets repeat every five minutes, pages repeat every twenty-four hours. The ticket queue floods; the page is forgotten. Symptom: ticket backlog fills the working hours; real incidents wait twenty-four hours for a repeat page.

How to troubleshoot it

When an alert has been routed to the wrong receiver, the order is:

1. Inspect the firing alert in the Alertmanager UI. Read the severity: label. Confirm whether the label is correct.
2. If the label is correct, the rule is correctly choosing page or ticket; the problem is upstream in the rule’s expression or threshold.
3. If the label is wrong, fix the label in the rule file and reload. Do not silence.
4. Re-run amtool config routes test with the corrected label set to confirm the route now resolves to the right receiver.
5. File a follow-up to add a unit test that pins the routing decision for the rule. The next PR review will catch the regression.

Security implications

The page channel reaches phones, pagers, and third-party incident tools. A page must not carry sensitive data in its annotations; a ticket can carry slightly more (the working-hours team is in an office with stronger access controls), but the rule of thumb is the same: no user-identifying data in the alert payload. The alert should interpolate only the labels the on-call needs to act.

PagerDuty / OpsGenie API keys are sensitive. The Alertmanager config must reference them through a secret store, not as plaintext. The service_key field in the example above is a placeholder; in production, mount the key from Vault or Kubernetes Secrets and reference it through an environment variable or a secret interpolation.

Performance implications

The Alertmanager evaluation path is in-memory and fast. The performance cost of routing is negligible compared to the rule evaluation in Prometheus. The performance cost of mis-routing is the operational cost described above: minutes of on-call time per false-positive page.

One performance-relevant note: Alertmanager deduplicates alerts by their label set. Two alerts with identical labels are deduplicated into a single notification. Use this to your advantage by including the discriminating labels (service, region, severity) in the alert rule’s labels block. Two alerts that should be one notification will become one; two alerts that should be separate notifications will not be deduplicated.

Production guidance

• The decision matrix is not optional. Document it in the team’s runbook repository; require it in the PR review template for new alerts.
• The default severity for a new rule is ticket. Promote to page only when the team has demonstrated the alert represents user-visible failure that requires action within the SLO mitigation window.
• The receiver choice must be visible in the rule file, not in Alertmanager configuration alone. The rule should carry the severity: label that determines the route; the Alertmanager config should match on that label.
• Pages and tickets have different repeat intervals for a reason. Do not set them the same.

Verification

You should now be able to answer:

• What three axes decide page or ticket, and what score does each axis contribute?
• What is the operational cost difference between a false-positive page and a missed incident, in minutes and error-budget terms?
• How does the repeat_interval: differ between pages and tickets in the Alertmanager route tree, and why?
• What command validates a route decision against the live config before a merge is deployed?

Quiz