Alert Canary

A team runs all three test tiers: unit test, synthetic series, end-to-end. Every CI run is green. Every nightly test passes. The production alerting chain is wired, the Alertmanager routing tree is mirrored from staging, the receiver accepts POSTs. Three months later, a real production outage occurs. The rule that should fire for the outage does not fire. The team investigates. The rule is loaded. The expression is correct. The for: dwell is satisfied. The metric is healthy. But the alert never arrives at Alertmanager. The reason: the production Alertmanager’s webhook URL was rotated six weeks earlier in a secret rotation; the staging webhook was not rotated; the staging test still passes because the staging URL is still valid; the production chain is broken silently.

The team had no canary. A canary is an alert that fires on a schedule in production. If the canary fires, the chain is alive. If the canary does not fire, the chain is broken. The discipline is to run a canary in production and to treat the absence of the canary page as a meta-alert.

What it is

An alert canary is a scheduled rule that exercises the alerting chain in production on a fixed cadence. The canary’s purpose is not to detect a real production incident; it is to detect that the alerting chain itself is alive. There are three common shapes:

  +---------------------------------------------+
  |  Shape 1: Scheduled test alert              |
  |    A rule that fires on a schedule          |
  |    (e.g., time() % 3600 == 0 fires once     |
  |     per hour). The rule exercises the full  |
  |    chain. If the alert fires, the chain is  |
  |    alive. If it does not fire on schedule,  |
  |    the chain is broken.                     |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Shape 2: Watchdog on a synthetic target    |
  |    A rule that fires when a synthetic       |
  |    target goes down                         |
  |    (e.g., synthetic_canary_up == 0). The     |
  |    synthetic target publishes a metric on    |
  |    a fixed interval; the rule fires when    |
  |    the metric goes to 0. If the rule fires, |
  |    the chain is alive. If the target is     |
  |    up and the rule fires, the rule's expr   |
  |    is broken. If the target is down and     |
  |    the rule does not fire, the chain is     |
  |    broken.                                  |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Shape 3: Meta-alert on alert staleness     |
  |    A rule that fires when the production    |
  |    Alertmanager has not evaluated alerts    |
  |    for a configured window                  |
  |    (e.g., time() - alertmanager_last_eval   |
  |     > 600). If the meta-alert fires,        |
  |    Alertmanager is stalled or down.         |
  +---------------------------------------------+

The three shapes are complementary. Shape 1 (scheduled test alert) confirms the chain is alive end-to-end. Shape 2 (watchdog on a synthetic target) confirms a specific link (the exporter scrape). Shape 3 (meta-alert on alert staleness) confirms Alertmanager itself is alive.

The right approach is to run all three shapes in production. The canary fires on a fixed cadence; the on-call rota expects the canary page. A missing canary page is itself a meta-alert that fires the on-call rota through a separate channel (a chat channel, an email, a second pager). The discipline is that the canary’s absence is as loud as the canary’s presence.

The most common shape is Shape 2: a watchdog on a synthetic target. The synthetic target publishes a metric that the rule watches. The rule fires when the metric goes to 0. The synthetic target is a small exporter that runs on a separate host with an independent view of the chain. The rule is a standard Prometheus alerting rule; the only difference from a production rule is the scoping label (job="synthetic-canary") that prevents the rule from matching real production series.

Why a sysadmin cares

The canary catches the chain-level failure shapes that the unit test, the synthetic series test, and the end-to-end test do not catch:

1. A rotated webhook URL in production. The staging webhook is not rotated; the staging test still passes; the production chain is broken silently. The canary fires on schedule in production; the webhook rejects the POST; the on-call rota does not receive the canary; the absence is the signal.
2. A dead Alertmanager. The production Alertmanager crashes or is paused; no alerts are delivered; the real outage goes undetected. The canary’s meta-alert on Alertmanager staleness fires when Alertmanager has not evaluated alerts for the configured window.
3. A dead route. The production routing tree is refactored; a route is deleted; the canary alert matches the deleted route; the alert is dropped. The absence of the canary page is the signal.

The canary is the only tier that catches these failures in production. The unit test catches wrong expressions. The synthetic series test catches live data drift. The end-to-end test catches the staging chain. The canary catches the production chain. The four tiers together cover the full failure surface.

How it works

The canary rule is a standard Prometheus alerting rule that fires on a fixed cadence. The rule’s expression references a synthetic series that the synthetic exporter publishes. The rule fires when the series goes to 0. The Alertmanager routes the canary to a dedicated receiver. The on-call rota expects the canary on schedule; a missing canary is the signal that the chain is broken.

  Synthetic exporter
        |
        |  publishes synthetic_canary_up{job="synthetic-canary"}
        |  on a fixed interval (15s default)
        v
  Prometheus scrape (production)
        |
        v
  Prometheus rule evaluation
        |
        |  Rule: SyntheticCanaryDown
        |  expr:  synthetic_canary_up{job="synthetic-canary"} == 0
        |  for:   5m
        v
  Alertmanager
        |
        |  Route match: test="synthetic-canary" receiver="canary-receiver"
        v
  Canary receiver (chat channel or low-priority pager)

The canary receiver is deliberately low-priority. The canary alert is not a production incident; it is a test. The on-call rota expects it on schedule and treats it as a confirmation that the chain is alive.

The synthetic exporter is a separate process that runs on a separate host. The exporter publishes synthetic_canary_up with value 1 when healthy and 0 when unhealthy. The test harness flips the value to 0 on a schedule (for example, every hour for 6 minutes) to simulate a failure. The rule fires; the chain exercises; the on-call rota receives the canary. After the dwell, the test harness flips the value back to 1; the rule clears.

The canary’s for: dwell is configured to match the test harness’s flip duration. If the test harness flips to 0 for 6 minutes, the rule’s for: is 5 minutes, so the rule fires for one minute before the test harness restores the value. The canary page arrives; the on-call rota acknowledges; the test harness restores the value; the rule clears.

How to configure it

The canary has three components: the synthetic exporter, the alert rule, and the Alertmanager route.

The synthetic exporter:

// observability/synthetic-canary/main.go
package main

import (
    "net/http"
    "sync"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    canaryUp = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "synthetic_canary_up",
        Help: "1 if the synthetic canary is healthy, 0 if not",
    })

    mu     sync.Mutex
    forced bool
)

func main() {
    canaryUp.Set(1)
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/admin/flip", func(w http.ResponseWriter, r *http.Request) {
        mu.Lock()
        defer mu.Unlock()
        if forced {
            canaryUp.Set(1)
            forced = false
        } else {
            canaryUp.Set(0)
            forced = true
        }
        w.WriteHeader(http.StatusOK)
    })

    // Schedule the flip every hour for 6 minutes
    go func() {
        for {
            time.Sleep(1 * time.Hour)
            mu.Lock()
            canaryUp.Set(0)
            forced = true
            mu.Unlock()
            time.Sleep(6 * time.Minute)
            mu.Lock()
            canaryUp.Set(1)
            forced = false
            mu.Unlock()
        }
    }()

    http.ListenAndServe(":9100", nil)
}

The exporter publishes the metric, exposes an admin endpoint for manual flipping, and runs a goroutine that flips the metric on a schedule. The schedule is hourly; the flip duration is 6 minutes; the rule’s for: dwell is 5 minutes.

The alert rule:

# observability/prometheus/rules/canary.yml
groups:
  - name: alert-pipeline-canary
    interval: 30s
    rules:
      - alert: SyntheticCanaryDown
        expr: synthetic_canary_up{job="synthetic-canary"} == 0
        for: 5m
        labels:
          severity: info
          team: observability
          test: synthetic-canary
        annotations:
          summary: 'synthetic canary is down (instance={{ $labels.instance }})'
          runbook_url: 'https://runbooks.example.com/observability/synthetic-canary'

      - alert: AlertPipelineStalled
        expr: |
          time() - alertmanager_last_evaluation_timestamp_seconds
            > 600
        for: 0m
        labels:
          severity: critical
          team: observability
          test: meta-alert
        annotations:
          summary: 'Alertmanager has not evaluated alerts in 10 minutes'
          runbook_url: 'https://runbooks.example.com/observability/alertmanager-stalled'

Two rules. The first (SyntheticCanaryDown) fires when the synthetic target is down. The second (AlertPipelineStalled) fires when Alertmanager has not evaluated alerts in 10 minutes — the meta-alert.

The Alertmanager route:

# observability/alertmanager/production/alertmanager.yml
route:
  receiver: default-receiver
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  routes:
    - matchers:
        - test = "synthetic-canary"
      receiver: canary-receiver
    - matchers:
        - test = "meta-alert"
      receiver: meta-alert-receiver
      # Meta-alerts go to a separate, high-priority receiver
      # because the absence of a canary is a production
      # failure shape.
      continue: true

receivers:
  - name: default-receiver
    # ... production receiver config ...
  - name: canary-receiver
    webhook_configs:
      - url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
        # Dedicated low-priority Slack channel
  - name: meta-alert-receiver
    pagerduty_configs:
      - service_key: 'XXXXX'
        # Meta-alerts page the on-call

The canary alerts go to a dedicated low-priority receiver. The meta-alerts (alert staleness) go to a high-priority receiver that pages the on-call. The continue: true clause ensures the meta-alert continues to be evaluated even if the canary route matches.

How to validate it

Three checks confirm the canary is wired correctly.

1. The canary fires on schedule.

amtool alert query alertname=SyntheticCanaryDown

Expected output, within 1 hour after the canary starts:

Alertname           State     Active Since
SyntheticCanaryDown firing    2026-08-13T03:00:00Z

A firing state on schedule confirms the canary is wired correctly.

2. The canary routes to the canary receiver.

amtool alert query alertname=SyntheticCanaryDown \
  | grep receiver

Expected output:

receiver: canary-receiver

A receiver field of canary-receiver confirms the routing tree routes the canary correctly.

3. The meta-alert fires when Alertmanager stalls.

Stop the production Alertmanager. Wait 10 minutes. Confirm the meta-alert fires:

amtool alert query alertname=AlertPipelineStalled

Expected output:

Alertname             State     Active Since
AlertPipelineStalled  firing    2026-08-13T03:10:00Z

A firing state confirms the meta-alert detects the stalled Alertmanager.

How it can fail

Six failure modes appear repeatedly when teams adopt the canary pattern.

1. The canary is on the same Prometheus as the production alerts. Symptom: the production Prometheus dies; the canary stops evaluating; the canary alert cannot fire; the on-call rota is not notified. Cause: the canary shares the failure domain. Fix: run the canary on a separate Prometheus with an independent view of the chain.
2. The canary is on the same Alertmanager as the production alerts. Symptom: the production Alertmanager dies; the canary alert cannot be delivered; the meta-alert cannot fire. Cause: the canary shares the failure domain. Fix: run the canary on a separate Alertmanager.
3. The canary’s for: dwell is too short. Symptom: the canary alert flaps every scrape interval when the synthetic briefly goes to 0. Cause: the dwell is too short to absorb a normal scrape gap. Fix: set the dwell to at least 5 minutes for a synthetic target scraped every 15 seconds.
4. The canary alert fires constantly. Symptom: the on-call rota is paged every hour for the canary alert. Cause: the canary’s severity is critical and the route does not filter on the test label. Fix: set the canary’s severity to info and route the canary to a dedicated low-priority receiver.
5. The canary’s metric name collides with a real metric. Symptom: a real production alert fires when the canary goes to 0. Cause: the synthetic exporter publishes a metric name that a real exporter also uses and the rule’s expr: does not filter on the synthetic label. Fix: prefix the metric with synthetic_canary_ and filter on job="synthetic-canary" in the rule.
6. The canary’s test harness does not restore the metric. Symptom: the canary fires once and never clears. Cause: the test harness crashed before restoring the metric to 1. Fix: the test harness must restore the metric in a trap or finally block, not in the happy path.

How to troubleshoot it

In order:

1. Is the canary firing? amtool alert query alertname=SyntheticCanaryDown. If the alert is not firing on schedule, the synthetic exporter is not publishing the metric, or the rule’s expr: does not match.
2. Is the canary routed correctly? amtool alert query alertname=SyntheticCanaryDown | grep receiver. The receiver field should be the canary receiver, not the default receiver.
3. Is the meta-alert firing? amtool alert query alertname=AlertPipelineStalled. If the meta-alert is firing, the production chain is broken; investigate Alertmanager.
4. Is the canary Prometheus healthy? curl -s http://canary-prometheus:9090/-/healthy. A non-200 response means the canary Prometheus is down; restart it.
5. Is the synthetic exporter publishing the metric? curl -s http://synthetic-canary:9100/metrics | grep synthetic_canary_up. An empty result means the exporter is down; restart it.

Security implications

• The canary receiver exposes the alerting chain’s health. A canary that pages for “Alertmanager stalled” reveals that Alertmanager exists and is monitored. Treat the canary’s existence as a discoverable artefact; restrict the canary’s annotations to non-sensitive runbook URLs.
• The synthetic exporter exposes a /admin/flip endpoint. This endpoint flips the metric and triggers the canary. The endpoint must be on a private network with no public ingress. A public endpoint lets an attacker force the canary to fire every minute.
• The meta-alert receiver pages the on-call. The meta-alert receiver must be on a separate channel from the production receiver so a misconfigured production route cannot suppress the meta-alert.

Performance implications

• The canary Prometheus scrapes one target every 15 seconds. The cost is one series per scrape. Trivial.
• The canary rule evaluates once per 30 seconds. The cost is one equality check on one series. Trivial.
• The canary Alertmanager receives one alert per hour. The cost is one alert per hour routed to a low-priority receiver. Trivial.
• The meta-alert rule evaluates once per 30 seconds. The cost is one subtraction check on one series. Trivial.

The total cost of the canary stack is a second Prometheus

• Alertmanager that handles one alert per hour. The benefit is that the alerting chain is self-aware. The trade-off is the right one for any team that depends on its alerting chain.

Production guidance

• Run the canary on a separate Prometheus and a separate Alertmanager. The canary must not share the failure domain it monitors.
• Set the canary’s severity to info and route to a dedicated receiver. The canary is not a production incident; it is a confirmation that the chain is alive.
• Wire the meta-alert to page the on-call. The absence of the canary is a production failure shape; the meta-alert must reach a high-priority receiver.
• Schedule the canary to fire on a fixed cadence. A canary that fires on every scrape interval is noise; a canary that fires once per hour is signal. Choose the cadence that balances signal and noise.
• Test the canary in staging before production. Run the end-to-end test against the canary in staging; confirm the canary fires, the route matches, and the receiver receives. Promote to production only after the staging test passes.
• Review the canary’s expected firing time in the runbook. The on-call rota must know when to expect the canary. A missing canary is the signal; the runbook names the expected cadence and the response.

Verification

You should now be able to answer:

• What are the three shapes of an alert canary, and what failure shape does each catch?
• Why must the canary run on a separate Prometheus and a separate Alertmanager from the production alerts?
• What is the meta-alert on alert staleness, and what failure shape does it catch?
• Why should the canary’s severity be info and route to a dedicated receiver?
• What is the most common shape of a canary in production, and how is the synthetic exporter wired?

Quiz