Alerting and SLOs: building meaningful on-call signals

Alerting is the most mis-done part of monitoring. Most teams have either too few alerts (problems discovered by users) or too many (on-call engineer has learned to ignore them). This lesson covers the SRE-inspired practices that produce alerts people actually act on.

Symptoms, not causes

The cardinal rule: alert on symptoms, not causes.

Bad: alert when CPU is >95%. CPU is a cause. Bad: alert when disk is >85%. Disk is a cause. Good: alert when API latency p99 is >500ms. Latency is a symptom. Good: alert when VM migrations fail. Failed migrations are a symptom.

A cause alert fires when there’s nothing wrong (CPU is high because of legitimate load). A symptom alert fires when users are affected.

The exception: cause alerts are useful for capacity planning (monthly review of trends), not for paging.

SLIs and SLOs

Service Level Indicator (SLI) is a metric that measures the service level from the user’s perspective:

• API availability: % of requests returning 200 in a time window
• Migration latency: time from “migrate now” to VM running on target node
• Backup success rate: % of scheduled backups that complete
• Storage latency: p99 of read/write operations

Service Level Objective (SLO) is the target for an SLI:

• API availability: 99.9% over 30 days
• Migration latency: under 60 seconds at p99
• Backup success rate: 99.5% over 30 days
• Storage latency: under 10 ms at p99

Error budget is 100% minus SLO. With 99.9% availability, the error budget is 0.1% — about 43 minutes per month. When the error budget is exhausted, the team stops shipping new features and focuses on reliability.

Defining SLOs for a PVE cluster

A reasonable starting set:

slos:
  api_availability:
    sli: availability of pveproxy 200 responses
    target: 99.9%      # 43 minutes of downtime per month
    window: 30d

  vm_lifecycle:
    sli: % of VM start/stop/migrate operations succeeding
    target: 99.5%      # some operations can fail due to user error
    window: 30d

  backup_success:
    sli: % of scheduled vzdump jobs completing
    target: 99%        # backup windows are tight
    window: 30d

  storage_latency:
    sli: p99 of disk read latency on local storage
    target: under 10ms
    window: 30d

These aren’t arbitrary — they’re negotiated with stakeholders (application owners, leadership, customers). The SLO is a promise the platform team makes; the error budget is the consequence of breaking it.

Burn-rate alerts

The modern alerting pattern: alert when error budget is being consumed faster than sustainable.

A 99.9% SLO over 30 days means 0.1% error budget = 43 minutes. “Burning” the budget at 1% (10x faster than sustainable) means you’re 30 days away from exhaustion. Burning at 10% means 3 days away.

Alert thresholds:

• 2% burn (alert after 1.5 hours of this rate) → 14 days to exhaustion
• 5% burn (alert after 36 minutes of this rate) → 6 days to exhaustion
• 10% burn (alert after 18 minutes of this rate) → 3 days to exhaustion

# 2% burn rate (14 day budget)
(
  1 - (
    sum(rate(pve_api_requests_total{status=~"2.."}[1h]))
    /
    sum(rate(pve_api_requests_total[1h]))
  )
) > (1 - 0.999) * 14 / 30

# 10% burn rate (3 day budget)
(
  1 - (
    sum(rate(pve_api_requests_total{status=~"2.."}[30m]))
    /
    sum(rate(pve_api_requests_total[30m]))
  )
) > (1 - 0.999) * 3

This pattern catches slow burns (which traditional alerts miss) and fast burns (which traditional alerts catch but page too late).

Alertmanager routing

Once an alert fires, it needs to go somewhere useful. Alertmanager handles routing, deduplication, grouping, and silencing.

# /etc/prometheus/alertmanager.yml
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: 'pagerduty-critical'
      group_wait: 10s

    - matchers:
        - severity="warning"
      receiver: 'slack-warnings'

    - matchers:
        - service="pbs"
      receiver: 'backup-team'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'REPLACE_ME'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/AAA/BBB/CCC'
      - channel: '#proxmox-alerts'

  - name: 'backup-team'
    email_configs:
      - to: 'backup-team@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.example.com:587'

Critical alerts page immediately. Warnings go to Slack. Service- specific alerts go to the right team.

Alert severity tiers

A tiered severity model:

Use these consistently. A “warning” that pages is a critical. An “info” that goes to Slack is a warning. Calibrate everyone on the team.

Reducing alert fatigue

The biggest threat to a healthy alerting system is noise. Symptoms of fatigue:

• Alerts that fire and resolve without anyone acting
• On-call engineer silencing alerts in bulk
• Alerts that fire and the on-call doesn’t know what to do

Fixes:

• Tune thresholds based on observed data, not docs. The default alertmanager thresholds are starting points, not destinations.
• Use SLO burn rates. Page only on symptoms that burn the error budget.
• Combine related alerts. 10 “disk usage > 80%” alerts are worse than 1 “cluster storage near capacity” alert.
• Silence the noise during planned work. Mark maintenance windows in alertmanager so deployment-time noise doesn’t page.
• Track alert volume per alert. If an alert fires 50 times a month, it’s misconfigured.

Alert review cadence

Weekly, review:

• New alerts (added this week)
• Top 10 alerts by frequency
• Alerts that fired and were resolved without action

Monthly, review:

• SLO achievement vs target
• Error budget burn rate
• On-call engineer satisfaction (yes/no for each alert)

Quarterly, review:

• SLOs themselves — are they still right?
• Alerting rules — are they causing the right actions?
• Incident post-mortems — did alerts catch this? Could they have caught it earlier?

Production considerations

• Alertmanager HA. Alertmanager itself is a single point of failure. Run two instances in HA mode with a virtual IP, or use Alertmanager cluster mode.
• Silencing windows. Maintenance windows are necessary but dangerous. Document every silence, and audit weekly.
• Routing trees. Routing rules can get complex. Test with a fake alert: does it reach the right channel? Does it silence correctly during a maintenance window?
• Alert write-back. When an alert is closed, log who closed it and why. This becomes audit data.

Key takeaways

• Alert on symptoms, not causes.
• Define SLIs and SLOs with stakeholders.
• Use burn-rate alerts to catch both fast and slow failures.
• Tune relentlessly. An alert no one acts on should not page.

Knowledge check