Synthetic Series for Alerts

A team wants to verify that its checkout error-rate alert fires correctly when the upstream payment service becomes slow. The alert rule is straightforward. The team cannot reproduce the production condition in staging — the staging payment service is healthy, the staging latency is low, and the test would have to wait for a real outage to drive the rule over threshold. The team has three choices: wait for a real outage, fabricate a manual incident, or inject a synthetic series that drives the rule over threshold under controlled conditions. The third choice is the synthetic series. It is the discipline of testing alerting without depending on production conditions.

What it is

A synthetic series is a Prometheus time series that does not come from a real production signal. It exists for one of two reasons: to drive an alert condition in a controlled test, or to verify that the alert chain is alive. The series can live in three places:

  +---------------------------------------------+
  |  Tier 1: Unit test fixture (input_series)   |
  |    - Lives in the promtool test rules       |
  |      fixture file                            |
  |    - Never touches a live Prometheus         |
  |    - Drives the rule in the in-memory        |
  |      rule manager                            |
  |    - Catches: wrong expression, wrong for:   |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Tier 2: Staging Prometheus injection       |
  |    - Synthetic exporter publishes metrics    |
  |      to a staging Prometheus                 |
  |    - Rule evaluates against the live TSDB    |
  |    - Catches: label drift, schema drift,     |
  |      missing joins, join-key mismatches      |
  +---------------------------------------------+
                     |
                     v
  +---------------------------------------------+
  |  Tier 4: Production canary series            |
  |    - Synthetic exporter publishes metrics    |
  |      to the production Prometheus            |
  |    - The series has a unique label           |
  |      (job=synthetic-canary) so real alerts   |
  |      never match                             |
  |    - The rule watches the canary label and   |
  |      alerts when the canary goes silent      |
  |    - Catches: dead chain in production       |
  +---------------------------------------------+

The synthetic series is the substrate of alert testing at every tier above the unit test. The unit test’s input_series: is itself a synthetic series, declared inline in the fixture and consumed by the in-memory storage. The staging injection is a synthetic series exposed by a test exporter and scraped by the staging Prometheus. The canary is a synthetic series exposed by a long-lived exporter and scraped by the production Prometheus.

Why a sysadmin cares

Production conditions are not reproducible. A rule that fires correctly on a real production outage cannot be verified in staging because the staging service never has the real outage. The only ways to test the rule are to wait for the real outage (unacceptable), fabricate a manual incident in staging (expensive, slow, flaky), or inject a synthetic series that drives the rule under controlled conditions.

The synthetic series turns alert testing into a deterministic exercise. The test author picks the value curve (0+1x10, a constant, a step function at a known timestamp). The rule evaluates against the curve. The expected alert state is asserted. The same test runs every day on a schedule. The result is a green or red signal that depends only on the rule expression and the synthetic series, not on whether the upstream service happens to be having a bad day.

The discipline pays off in two currencies:

• Faster feedback on rule changes. A rule author can change a threshold, run the synthetic test, and see the result in seconds. The change can be reviewed before it reaches production.
• Continuous verification. The canary runs every minute in production. If the rule is dead, the canary fires. If the canary does not fire, the rule is dead. The on-call rota trusts the alert because the alert is verified every minute.

How it works

The synthetic series flows through the same Prometheus path as a real series, with one difference: the source is a test exporter or a Pushgateway push, not a real production service.

  Synthetic exporter OR Pushgateway push
        |
        |  HTTP GET /metrics OR
        |  POST /metrics/job/...
        v
  Prometheus scrape (production or staging)
        |
        |  Series: synthetic_canary_up{job="synthetic-canary",
        |           instance="canary-1"} 0|1
        v
  Prometheus rule evaluation
        |
        |  Rule: alert:SyntheticCanaryDown
        |  expr:  synthetic_canary_up == 0
        |  for:   5m
        v
  Alertmanager
        |
        v
  Receiver (Slack, PagerDuty, webhook)

The synthetic series has the same shape as a real series: metric name, labels, values. The only thing that makes it synthetic is the source. The rule does not know (and does not need to know) whether the series comes from a real service or a test exporter.

The right approach is to scope the synthetic series to a dedicated job label (job="synthetic-canary" or source="synthetic"). The rule’s expr: includes the scoping label, so the rule only fires for the synthetic series. A real production series cannot accidentally match the rule’s selector because the production series does not carry the synthetic label. The discipline of scoping is the single most important property of a synthetic series for alert testing.

How to configure it

The most common shape is a synthetic exporter that publishes a single metric under a dedicated job label. The rule watches the metric and alerts when it deviates. The rule’s expr: includes the synthetic label so it cannot fire on a real signal.

The synthetic exporter (a small Go HTTP service):

// observability/synthetic-canary/main.go
package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    canaryUp = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "synthetic_canary_up",
        Help: "1 if the synthetic canary is healthy, 0 if not",
    })
)

func main() {
    canaryUp.Set(1) // healthy on start
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":9100", nil)
}

The exporter is a static binary with one metric. The test harness flips the metric to 0 to simulate a failure, then flips it back to 1. The rule fires when the metric is 0 for the configured dwell.

The Prometheus scrape config (production):

# observability/prometheus/prometheus.yml
scrape_configs:
  - job_name: synthetic-canary
    static_configs:
      - targets: ['synthetic-canary:9100']
    scrape_interval: 15s
    metrics_path: /metrics

The job name synthetic-canary is the scoping label. Every metric from this scrape carries job="synthetic-canary". The rule’s expr: filters on this label.

The alert rule:

# observability/prometheus/rules/canary.yml
groups:
  - name: alert-pipeline-canary
    interval: 30s
    rules:
      - alert: SyntheticCanaryDown
        expr: synthetic_canary_up{job="synthetic-canary"} == 0
        for: 5m
        labels:
          severity: warning
          team: observability
          test: synthetic-canary
        annotations:
          summary: 'synthetic canary is down (instance={{ $labels.instance }})'
          runbook_url: 'https://runbooks.example.com/observability/synthetic-canary'

The expr: filters on job="synthetic-canary". A real production series cannot match because no real production service carries that label. The test: synthetic-canary label on the alert marks it as a test alert so the on-call rota knows to ignore it.

The test harness (drives the synthetic):

#!/usr/bin/env bash
# observability/synthetic-canary/test.sh
set -euo pipefail

CANARY="http://synthetic-canary:9100"

echo "Driving canary to 0"
curl -s -X POST "${CANARY}/admin/down"

echo "Waiting 6 minutes for alert to fire"
sleep 360

ALERT_STATE=$(amtool alert query alertname=SyntheticCanaryDown \
  | awk '/firing/ {print $3; exit}')

if [ "$ALERT_STATE" != "firing" ]; then
    echo "FAIL: alert did not fire"
    exit 1
fi

echo "Restoring canary to 1"
curl -s -X POST "${CANARY}/admin/up"

echo "PASS"

The test harness flips the metric, waits for the rule’s for: dwell plus one scrape interval, asserts the alert fired, then restores the metric. The script runs nightly in CI; a failure posts to the observability chat channel.

How to validate it

Three checks confirm the synthetic series is scoped correctly and the rule fires as expected.

1. The synthetic series carries the scoping label.

curl -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=synthetic_canary_up' \
  | jq '.data.result[].metric'

Expected output:

{
  "__name__": "synthetic_canary_up",
  "instance": "synthetic-canary:9100",
  "job": "synthetic-canary"
}

A missing job label means the scrape config dropped the job name. The fix is to restore the job_name in the scrape config.

2. The rule fires when the synthetic is at 0.

curl -s -X POST http://synthetic-canary:9100/admin/down
sleep 360
amtool alert query alertname=SyntheticCanaryDown

Expected output, after 6 minutes:

Alertname           State     Active Since
SyntheticCanaryDown firing    2026-08-13T03:00:00Z

A firing state confirms the rule evaluates correctly against the live TSDB.

3. The alert does not fire when the synthetic is at 1.

curl -s -X POST http://synthetic-canary:9100/admin/up
sleep 60
amtool alert query alertname=SyntheticCanaryDown

Expected output, after 1 minute:

Alertname           State
(no alerts)

An empty result confirms the rule clears when the synthetic restores. If the alert remains firing, the rule’s for: dwell is broken or the rule has a sticky-state bug.

How it can fail

Six failure modes appear repeatedly when teams adopt the synthetic series pattern.

1. The synthetic label is missing. Symptom: the rule never fires because no series matches the selector. Cause: the scrape config dropped the job_name or the static config renamed the target. Fix: restore the label and assert it in the test fixture.
2. The synthetic series leaks into real alerts. Symptom: a real production alert fires when the synthetic goes to 0. Cause: the rule’s expr: does not filter on the synthetic label. Fix: add the label filter to the rule’s expr:.
3. The synthetic exporter’s metric name collides with a real metric. Symptom: dashboards show two series with the same name, one from the synthetic, one from the real service. Cause: the synthetic exports under a name that a real exporter also uses. Fix: prefix the synthetic metric with synthetic_canary_ and the scrape job with synthetic-canary.
4. The Pushgateway synthetic outlives the test. Symptom: a test pushes a metric to the Pushgateway and the test fails before the metric is deleted; the metric persists for hours and the alert fires for hours. Cause: the test harness does not clean up. Fix: always delete the metric from the Pushgateway at the end of the test (use the Pushgateway’s delete API, not the push API).
5. The synthetic series has a high cardinality label. Symptom: Prometheus ingests thousands of synthetic series because the test exporter publishes one series per instance per test_id per run_id. Cause: the test harness adds a unique label per run. Fix: keep the label set fixed; use the metric value to vary the behaviour, not the labels.
6. The synthetic is on the same Prometheus that evaluates the alert. Symptom: the Prometheus dies; the synthetic stops scraping; the rule stops evaluating; the alert does not fire; the failure is silent. Cause: the synthetic shares the failure domain. Fix: run the synthetic on a separate Prometheus with an independent view of the chain, or alert on up == 0 for the synthetic exporter’s job (a meta-alert on the meta-alert).

How to troubleshoot it

In order:

1. Does the series exist? curl -s http://prometheus:9090/api/v1/query --data-urlencode 'query=synthetic_canary_up' | jq. An empty result means the scrape is not happening; check the targets page (/targets).
2. Does the series carry the scoping label? The jq output above should show job="synthetic-canary". A missing label means the scrape config dropped the job_name.
3. Does the rule evaluate? curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name=="SyntheticCanaryDown") | \{health, lastError\}'. A non-empty lastError means the rule loaded but failed to evaluate; investigate the TSDB state.
4. Does the alert fire? amtool alert query alertname=SyntheticCanaryDown. If the alert is not firing after the dwell, the rule’s expr: does not match the synthetic series; check the label selector.
5. Does the receiver receive? Check the Alertmanager log for the receiver delivery; check the receiver’s access log for the incoming POST.

Security implications

• The synthetic exporter exposes a /admin/down endpoint. This endpoint flips the metric to 0 and triggers the alert. The endpoint must be on a private network, behind a firewall, or behind an authentication layer. A public endpoint lets an attacker force the on-call rota to receive a synthetic page every minute.
• The synthetic exporter’s metric name reveals the test’s existence. A metric named synthetic_canary_up discloses that the team runs a synthetic canary alert. Treat the metric name as discoverable; restrict the Alertmanager annotations to non-sensitive runbook URLs.
• The Pushgateway synthetic is push-based, not pull-based. An attacker with network access to the Pushgateway can push arbitrary metrics that drive the alert. Restrict the Pushgateway to the test harness’s network.

Performance implications

• The synthetic exporter is one scrape per interval. A 15-second scrape interval means four scrapes per minute. The exporter serves one metric (synthetic_canary_up). The cost on Prometheus is one series per scrape. Trivial.
• The rule evaluates once per interval. A 30-second interval means two evaluations per minute. The expr: is a single equality check on a single series. Trivial.
• The Pushgateway synthetic accumulates stale metrics. A Pushgateway push persists until the metric is explicitly deleted. A test that pushes a metric and crashes before deleting it leaves the metric in the Pushgateway for hours. The Prometheus scrape picks up the stale metric and the alert fires indefinitely. Clean up after every push.

Production guidance

• Always scope the synthetic by label. The scrape config sets the job_name; the rule’s expr: filters on it; the alert’s test: label marks it as a test. The label is the wall.
• Use the exporter pattern, not the Pushgateway, for continuous canaries. The exporter is pull-based; the Pushgateway is push-based and accumulates stale metrics. Use the Pushgateway only for short-lived test pushes with explicit cleanup.
• Assert the label set in the unit test. The unit test fixture for the canary rule should include a test that asserts the synthetic label is required; a fixture that omits the label fails when the rule is refactored to remove the label filter.
• Run the synthetic on a separate Prometheus for the production canary. The canary’s purpose is to monitor the chain; if the canary shares the chain’s failure domain, the canary cannot fire when the chain breaks.
• Clean up Pushgateway metrics at the end of every test. A test harness that pushes a metric must delete the metric in a trap or finally block, not in the happy path. The Pushgateway has no TTL.

Verification

You should now be able to answer:

• What are the three places a synthetic series can live, and which failure shape does each catch?
• Why must the synthetic series be scoped by label, and what is the wall between the test and the production alert path?
• What is the most common shape of a synthetic series for alert testing, and what exporter pattern does it use?
• Why must a Pushgateway-based synthetic always clean up after the test, and what happens if it does not?
• Why should the production canary’s synthetic series live on a Prometheus that is not the Prometheus it monitors?

Quiz