Business Alerting

The on-call engineer is paged at 03:14 on a Sunday. The alert is BusinessRevenueDown: the orders completed counter has fallen by more than 30 percent in the last 5 minutes. The engineer opens the dashboard. The technical RED metrics are at 100 percent availability. The funnel counter is at 105 percent of the same-hour-last-week. The orders completed counter is a false positive: the alert is firing on a 5-minute window that happened to coincide with a number being reset by a backfill job that ran for 4 minutes at 03:10.

The engineer silences the alert; the next alert is the same false positive at 03:40. By 04:00 the engineer has silenced the alert for the night. By 09:00 the next day the team has deleted the alert because “it fires too often”. By 14:00 on Monday the business asks why the conversion fell off a cliff on Sunday afternoon. The team has no answer because the alert is disabled.

This is the cost of a business alert that fires on the wrong signal. The team that has the right thresholds and the right routing is the team that catches the real failures and ignores the noise.

What it is

Business alerting is the discipline of paging on the business counters exactly when the business impact is real, exactly when the on-call engineer can do something about it, and never otherwise. The discipline has three rules:

1. Sharp drop = page. A 30 percent fall in the conversion rate in 5 minutes is a sharp drop. The on-call engineer can investigate. The alert pages.
2. Slow drift = ticket. A 5 percent fall in the conversion rate over 7 days is a slow drift. The on-call engineer cannot fix the drift in a single shift. The alert opens a ticket in the product backlog.
3. Statistical noise = panel. A 0.5 percent oscillation in the conversion rate is statistical noise. The alert is a dashboard panel, not an alert at all.

The distinction is the threshold type and the for: (action) distinction. A page implies a 5-minute response; a ticket implies a 14-day response; a panel implies no automated response. The threshold and the window must match the action.

  Signal type               Threshold               Window    Action
  ------------------------  ----------------------  --------  --------
  Sharp drop                rate of change > 30%    5 min     page
  Slow drift                absolute level < X      7 days    ticket
  Statistical noise         (no threshold)          -         panel
  Sustained outage          rate == 0               1 min     page
  Stale counter             absent()                15 min    page

The right answer is fewer pages, more panels, and a clear ticket lane for slow drift. The team that has three pages a month on the business counter is the team that has the discipline.

Why a sysadmin cares

Three reasons the operator cares about business alerting:

1. The business alert is the operator’s relationship with the executive team. The executive team reads the page, not the panel. The alert that pages is the alert that defines the operator’s reputation.
2. The business alert is the operator’s last line of defence. The technical alert catches the technical failure. The business alert catches the failure that the technical alert missed: the user-visible failure that the technical metrics were green for.
3. The business alert is the operator’s discipline. The team that has the right thresholds and the right routing has the discipline. The team that pages on every counter movement has lost the discipline.

None of this is the operator’s job description. All of it is the operator’s actual job.

How it works

The alerting model is a chain of three components: the Prometheus alert rule, the Alertmanager routing policy, and the notification target. The discipline lives in the threshold, the window, and the route.

  Prometheus alert rule
  --------------------------
  threshold + window + for:
    sharp drop:  rate < 0.7 * rate[1h]  for 5m
    slow drift:  rate < 0.95 * rate[7d] for 1d
    stale:       absent()               for 15m
                                                    |
                                                    v
  Alertmanager routing policy
  --------------------------
  matchers + receivers:
    severity=page    -> pager team
    severity=ticket  -> service desk
    severity=panel   -> no route (panel only)
                                                    |
                                                    v
  Notification target
  --------------------------
  page  -> PagerDuty / Opsgenie (5-minute response)
  ticket -> Jira / Linear (14-day response)

The discipline is the boundary between page and ticket. The boundary is the action, not the threshold. A page implies something the operator can do in 5 minutes; a ticket implies something the operator cannot.

How to configure it

The configuration is three Prometheus alert rules and one Alertmanager routing policy.

# /etc/prometheus/rules/business_alerts.yml
groups:
  - name: business_alerts
    interval: 30s
    rules:
      # Sharp drop: orders completed falls by more than 30 percent
      # compared to the same hour last week, sustained for 5 minutes.
      - alert: OrdersCompletedSharpDrop
        expr: |
          sum(rate(orders_completed_total[5m]))
            <
          0.7 * sum(rate(orders_completed_total[1h] offset 7d))
        for: 5m
        labels:
          severity: page
          team: business-on-call
        annotations:
          summary: "Orders completed has fallen 30% week-over-week"
          description: |
            The orders completed rate has fallen by more than 30 percent
            compared to the same hour last week. Sustained for 5 minutes.
            Check the funnel dashboard and the technical RED metrics.

      # Slow drift: conversion rate falling 5 percent over 7 days.
      - alert: ConversionRateSlowDrift
        expr: |
          avg_over_time(
            sum(rate(orders_completed_total[5m]))
            /
            sum(rate(orders_started_total[5m]))[7d]
          )
          <
          0.95 * avg_over_time(
            sum(rate(orders_completed_total[5m]))
            /
            sum(rate(orders_started_total[5m]))[7d] offset 7d
          )
        for: 1d
        labels:
          severity: ticket
          team: growth
        annotations:
          summary: "Conversion rate has drifted 5% over the last week"
          description: |
            The conversion rate has fallen by more than 5 percent over
            the last 7 days, compared to the previous 7 days. Open a
            ticket in the growth backlog.

      # Stale counter: the business pipeline is silent.
      - alert: BusinessCounterStale
        expr: absent(rate(orders_completed_total[5m]))
        for: 15m
        labels:
          severity: page
          team: business-on-call
        annotations:
          summary: "Orders completed counter is silent"
          description: |
            The orders completed counter has not been updated for 15
            minutes. The collector pipeline or the upstream source is
            down. Page.

The Alertmanager routing policy:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [alertname, severity]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: page
      receiver: pagerduty-business
      group_wait: 30s
      group_interval: 1m
      repeat_interval: 1h
    - match:
        severity: ticket
      receiver: jira-business-backlog
      group_wait: 5m
      group_interval: 1h
      repeat_interval: 24h
receivers:
  - name: pagerduty-business
    pagerduty_configs:
      - service_key: "<pagerduty-service-key>"
        description: "{{ .CommonAnnotations.summary }}"
  - name: jira-business-backlog
    webhook_configs:
      - url: "https://jira.example.com/webhook/business"
        send_resolved: true
  - name: default
    webhook_configs:
      - url: "https://slack.example.com/webhook/alerts"

The boundary is the route. The severity: page label is the page; the severity: ticket label is the ticket. The Alertmanager applies the rule.

How to validate it

# Is the alert rule loaded?
curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | select(.name == "business_alerts") | .rules[] | .name'
# Expected: three rule names

# Can the alert be synthesised?
# Simulate a sharp drop by writing a low value to the counter
# (only in staging).
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(orders_completed_total[5m]))' \
  | jq '.data.result[0].value[1]'
# Expected: a low value to trigger the rule

# Is Alertmanager receiving the alert?
amtool alert query --alertmanager.url=http://localhost:9093 \
  --team business-on-call
# Expected: the active alert

# Is the route correct?
amtool config show --alertmanager.url=http://localhost:9093 \
  | jq '.route.routes[] | select(.match.severity == "page") | .receiver'
# Expected: pagerduty-business

# Is the stale counter alert evaluated?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=absent(rate(orders_completed_total[5m]))' \
  | jq '.data.result'
# Expected: an empty array when the counter is alive

How it can fail

1. The alert fires on the backfill job. Symptom: the counter is briefly reset by a backfill; the alert fires on the 5-minute window. The fix is to mark the backfill window with a maintenance label or to use the offset modifier on the baseline.
2. The alert fires on the weekly seasonality. Symptom: the counter drops every Sunday at 03:00 because the business is closed. The fix is to use offset 7d on the baseline so the comparison is the same hour last week.
3. The alert is silenced because it fires too often. Symptom: the on-call engineer silences the alert for the night; the alert is then deleted. The real failure is missed. The fix is to fix the threshold before the alert is added.
4. The slow drift alert is a page. Symptom: the page reaches the on-call engineer; the engineer cannot fix the drift in a single shift. The engineer silences the alert. The fix is to route the slow drift to a ticket, not a page.
5. The stale counter alert is missing. Symptom: the collector is down for 30 minutes; no alert fires because the stale counter rule was never written. The fix is to add the absent() rule.
6. The Alertmanager route is wrong. Symptom: the page goes to the ticket lane; the ticket goes to the pager. The fix is to verify the routes with amtool config show before the alert is added.

How to troubleshoot it

1. Is the alert rule loaded? Check /api/v1/rules for the business_alerts group.
2. Is the route in Alertmanager correct? Run amtool config show against the live Alertmanager.
3. Is the alert firing for the right reason? Read the alert annotations; check the time window of the violation.
4. Is the alert silenced? amtool silence query for the alert name.
5. Is the threshold right? Compare the threshold to the last 30 days of the metric. The threshold should fire roughly once per month, not once per day.

Security implications

• PagerDuty credentials. The Alertmanager PagerDuty service key is a write credential to the on-call rotation. The key should be in a secrets manager, not in the config file.
• Jira webhook. The webhook URL is a write credential to the Jira backlog. The URL should be in a secrets manager and scoped to the business backlog only.
• Slack webhook. The webhook URL is a write credential to the alert channel. The URL should be in a secrets manager and scoped to the alert channel.

Performance implications

The performance cost of a business alert is dominated by the rule evaluation cost, not the Alertmanager routing. The rule evaluator runs every 30 seconds; the expression is a constant time operation for a small number of series. The cost is trivial.

The trade-off is the alert volume. A team that pages on every 5 percent move has 30 pages a month. The team that pages on the 30 percent move has 3 pages a month. The right answer is the threshold that matches the action.

Verification

You should now be able to answer:

• What is the boundary between a page and a ticket in business alerting?
• Why should the offset 7d modifier be used on the baseline for sharp-drop alerts?
• What is the absent() rule for, and why is it the most important business alert?
• How does the Alertmanager routing policy enforce the page / ticket boundary?
• What is the first symptom that the business alert is misconfigured?

Quiz