Synthetic Cost and Where to Run

A team builds a synthetic suite to catch the regional DNS outage that paged them last quarter. The suite has 800 probes at 15-second intervals from three regions. The dashboard is green. The team’s Prometheus is on its knees: the exporter is using six gigabytes of RAM, the scrape budget is exhausted, and the TSDB has 1.5 million active series dedicated to probe metrics. The application team files a ticket: “your probes are slowing our application.” The synthetic suite, built to detect outages, is now causing one.

Cost is not a footnote of synthetic monitoring. Cost is the operational lever that decides whether the suite earns its keep.

What it is

Synthetic cost is the budget impact of running probes: exporter CPU and RAM, network egress from the probe locations, scrape concurrency on Prometheus, TSDB head series and on-disk storage, and the load the probes place on the target services. The right suite is right-sized: it catches the failure modes the team cares about, at a cadence the SLA demands, without exceeding the budgets the platform and the targets can sustain.

The four budget categories are independent. A suite that fits the scrape budget can still exhaust the target’s capacity. A suite that fits the exporter can still exhaust the TSDB. The disciplines that bound each budget are different.

Why a sysadmin cares

A synthetic suite that exceeds the platform’s budgets becomes a liability. Five production consequences of an over-built suite:

• The suite is the outage. Probes at sub-minute cadence against a low-traffic application can become the dominant traffic source. The suite detects an outage by causing it.
• The suite exhausts the scrape budget. Prometheus’s global concurrency limit is shared with every other scrape job. A synthetic suite with 1 000 targets at 15 s is a 67 rps load; the rest of the platform loses its scrape budget.
• The suite exhausts the TSDB. Each probe target emits roughly 10 active series. A 1 000-target suite is 10 000 active series; a 10 000-target suite is 100 000. The TSDB’s head-series budget is consumed by probe metrics that are dashboard-only.
• The suite’s on-call bill. Probes that test decommissioned services, internal hosts that no longer exist, or services that have moved are still paged. The on-call bill grows with the suite, not with the value.
• The suite’s egress bill. External vantage points (third-party probe providers) charge per probe or per probe-minute. A 200-target suite at 60 s intervals across three regions is roughly 8.6 million probe-minutes per year. The line item is real.

The right suite is the suite that earns its keep: catches the failure modes the team has actually paged on, at a cadence that fits the SLA, without exceeding any of the four budgets.

How it works

The cost model has four categories. Each category is bounded by a different discipline.

   exporter CPU + RAM        scrape concurrency          TSDB head series
   -------------------       -----------------          -----------------
   bounded by:               bounded by:                 bounded by:
   - module type             - global concurrency limit  - relabel discipline
   - timeout                 - scrape_interval           - cardinality budget
   - target count            - target count              - retention window
   - vantage point count     - scrape_timeout

        |                          |                          |
        +-------------+------------+----------+---------------+
                      |                       |
                      v                       v
              network egress            target load
              --------------            ------------
              bounded by:               bounded by:
              - cadence                 - cadence
              - target count            - target count
              - vantage point count     - vantage point count
              - payload size            - payload size

The exporter CPU is the cheapest budget to exhaust. blackbox_exporter is a single Go process; it scales linearly with the per-second probe rate, not the target count. A 200-rps probe rate saturates a modern four-core exporter. The exporter’s RAM grows with the number of concurrent probes (one goroutine per probe).

The scrape concurrency is bounded by Prometheus’s --query.lookback-delta and the scrape queue depth. The right discipline is to size the synthetic suite’s target count and cadence so that the synthetic suite occupies a predictable fraction of the global scrape budget.

The TSDB head series is bounded by the relabel discipline. Each probe target emits roughly 10 active series. A 1 000-target suite is 10 000 active series; a 10 000-target suite is 100 000. The discipline is to label by service, not by URL; to drop unneeded labels; and to retire probes that test decommissioned services.

The target load is bounded by the cadence and the target’s capacity. A 200-target suite at 60 s intervals against a low-traffic application is roughly 3.3 rps inbound to the application. A 1 000-target suite at 15 s intervals is 67 rps — a non-trivial fraction of a small application’s traffic.

How to configure it

The right configuration is the one that right-sizes the suite. Below is a production-shaped scrape configuration that documents the cadence and the budgets.

# /etc/prometheus/prometheus.yml
scrape_configs:
  # Cheap HTTP boundary probes. High-frequency because
  # the boundary is the highest-value assertion.
  - job_name: blackbox_http_boundary
    metrics_path: /probe
    params:
      module: [http_2xx_boundary]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - https://example.com/healthz
          - https://api.example.com/v1/ping
        labels:
          service: public-boundary
          tier: critical
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(https?://[^/]+)'
        replacement: '${1}'
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

  # TLS probes against the same targets. Lower-frequency
  # because the cert expiry is the value, not the per-second
  # health.
  - job_name: blackbox_tls_public
    metrics_path: /probe
    params:
      module: [http_2xx_https_strict]
    scrape_interval: 300s        # 5 minutes is enough
    scrape_timeout: 15s
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com
        labels:
          service: public-boundary
          tier: critical
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(https?://[^/]+)'
        replacement: '${1}'
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

  # DNS probes against the operator's recursive. Low
  # frequency because the resolver is the value, not the
  # per-second lookup.
  - job_name: blackbox_dns
    metrics_path: /probe
    params:
      module: [dns_recursive]
    scrape_interval: 60s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - api.example.com
        labels:
          service: api-public
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

  # Multi-step journeys. Low frequency because the
  # journey is expensive and the SLA on the journey
  # is minutes, not seconds.
  - job_name: pushgateway_journey
    honor_labels: true
    scrape_interval: 60s
    scrape_timeout: 10s
    static_configs:
      - targets: ['pushgateway.internal:9091']
        labels:
          service: checkout-journey
          env: prod

The right cadence is the SLA minus the journey cost. The boundary probe runs at 30 s because the boundary SLA is 30 s. The TLS probe runs at 5 minutes because the cert expiry is days, not seconds. The DNS probe runs at 60 s because the resolver SLA is 60 s. The journey runs on a 5-minute cron because the journey cost is 4 s and the flow SLA is 5 minutes.

How to validate it

# 1. Calculate the per-second probe rate for each job.
echo "http_boundary: $(($(wc -l < /etc/blackbox/targets/http_boundary.txt) / 30)) rps"
echo "tls_public:    $(($(wc -l < /etc/blackbox/targets/tls_public.txt) / 300)) rps"
echo "dns:           $(($(wc -l < /etc/blackbox/targets/dns.txt) / 60)) rps"

# 2. Calculate the TSDB head series contribution.
curl -s http://prometheus.internal:9090/api/v1/status/tsdb \
  | jq '.data.headSeriesValueByLabelName // .data.seriesCountByMetricName' | head
# probe_success: 412
# probe_duration_seconds: 412
# probe_http_status_code: 412
# probe_ssl_earliest_cert_expiry: 200
# ...

# 3. Calculate the active series per job.
curl -sg http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=count by (job) (probe_success)' \
  | jq '.data.result[] | {job: .metric.job, count: .value[1]}'
# {"job":"blackbox_http_boundary","count":"412"}
# {"job":"blackbox_tls_public","count":"200"}
# {"job":"blackbox_dns","count":"50"}

# 4. Confirm the exporter is not saturating.
curl -s http://blackbox.internal:9115/-/healthy
# Prometheus Blackbox Exporter is Healthy.
ps -o pid,rss,cmd -p $(pgrep blackbox_exporter)
# 12345  412000 blackbox_exporter --config.file=...

# 5. Confirm the egress from the probe host.
iftop -t -s 30 -n -P 2>/dev/null | grep "203.0.113" | head
# ...

A green exporter, a sane active-series count, and a sane egress rate are the headline assertions. The right suite is one where each of those numbers fits the budget the platform has set.

How it can fail

1. Cadence exceeds the target’s capacity. A 200-target suite at 15 s against a low-traffic application. The suite’s traffic is the application’s dominant load. The application slows down; the probes slow down; the probes report the slowdown they caused. Symptom: probe_duration_seconds rises with the cadence; the application logs show probe traffic.

2. Scrape budget exhaustion. A suite with many scrape jobs and many targets occupies most of the global scrape concurrency. The rest of the platform’s scrapes queue behind the synthetic suite. Symptom: dashboards for non-synthetic metrics show intermittent gaps.

3. TSDB head series exhaustion. A 10 000-target suite at 10 active series per target is 100 000 active series. The TSDB’s head-series budget is consumed by probe metrics. Symptom: Prometheus logs head series out of bounds; ingest of other metrics is paused.

4. External vantage-point bill. A 200-target suite at 60 s intervals across three external regions is roughly 8.6 million probe-minutes per year. At a typical per-probe-minute price, the line item is real. Symptom: the team’s quarterly review asks why the observability budget grew 30%.

5. The suite is the longest tail of unused probes. Probes that test decommissioned services, internal hosts that no longer exist, or services that have moved are still scraped. The TSDB stores their metrics. Symptom: dashboard shows green for services that don’t exist; the suite’s cardinality grows unboundedly.

6. Browser journey cost spike. A Playwright journey that upgrades to a heavier browser. Each journey takes 8 s instead of 4 s. The runner exhausts its concurrency budget. Symptom: journeys queue; the on-call sees “checkout journey is slow” alerts.

7. Goroutine leak during a regional brownout. A sustained brownout in one region. The exporter holds goroutines for the duration of each slow probe. The exporter’s RAM grows linearly. Symptom: exporter OOMs mid-brownout; the very failure the suite was meant to detect.

How to troubleshoot it

The order matters because the boundary at which the cost overshoot lives determines the remedy.

1. Is the suite within the active-series budget? count by (job) (probe_success). If the synthetic suite exceeds the budget, retire probes or lengthen the cadence.
2. Is the suite within the scrape concurrency budget? Inspect Prometheus’s scrape queue depth and the --storage.tsdb.head-chunks-write-queue-size. If the synthetic suite dominates the queue, consolidate scrape jobs.
3. Is the suite within the exporter’s capacity? ps -o rss -p $(pgref blackbox_exporter). A four-core exporter sustains roughly 200 rps; beyond that, scale the exporter horizontally.
4. Is the suite within the target’s capacity? Compare the per-second probe rate to the target’s request rate. If the probe rate exceeds 5% of the target’s request rate, the cadence is too aggressive.
5. Is the suite within the egress budget? iftop or the cloud provider’s egress dashboard. If the probe network egress exceeds the budget, retire probes or consolidate vantage points.
6. Is the suite still earning its keep? Inventory the probes; retire the ones that test decommissioned services.

Security implications

The exporter accepts an arbitrary target as a URL parameter. A suite that exposes the exporter without authentication is a port-scan primitive. The right discipline is to bind the exporter to a private network and to authenticate the scrape from Prometheus.

External vantage points charge per probe or per probe-minute. A credential that is shared across vantage-point providers is a credential that the provider’s operators can read. The right discipline is to use a managed-probe credential scoped to the specific vantage-point tenant, rotated on the same cadence as other platform credentials.

The journey runner holds credentials: a session cookie, an API token, a payment instrument. Those credentials must be stored in a secret manager and rotated regularly. A journey that triggers rate limits on a downstream provider is a journey that pages the on-call for the wrong reason.

Performance implications

The performance implications of an over-built suite are the failure modes above. The right suite is one that respects each of the four budgets.

• Exporter capacity. Roughly 200 rps on a four-core exporter. Scale horizontally beyond that.
• Scrape concurrency. Set scrape_timeout to 2 * scrape_interval. Consolidate to a small number of scrape jobs.
• TSDB head series. Roughly 10 active series per probe target. Budget the suite by the TSDB’s head-series limit.
• Target capacity. Probe rate should not exceed ~5% of the target’s request rate for low-traffic applications.
• Network egress. Set a per-vantage-point egress budget; retire probes that exceed it.

Production guidance

• Pick the cadence deliberately. The right cadence is the SLA minus the cost. The boundary probe is the cheapest signal; it runs at 30 s. The cert-expiry probe is the most patient signal; it runs at 5 minutes. The journey is the most expensive signal; it runs at 5-minute to 30-minute cadence.
• Pick the vantage point deliberately. At least two regions; one external vantage point if the budget allows. The internal vantage point is the cheapest; the external vantage point is the most informative.
• Document the budget. Active series per job, exporter RAM, scrape concurrency, target load, and egress. Review quarterly.
• Retire probes that test decommissioned services. An inventory is the operational discipline that keeps the suite right-sized.
• Consolidate scrape jobs. A small number of scrape jobs with predictable cadence is easier to budget than many scrape jobs with varying cadence.
• Negotiate synthetic carve-outs with downstream providers. The journey’s traffic is real traffic; the provider’s rate limit is the boundary.
• Treat the journey runner as code. The journey is a load test against the production path; review it.

Verification

You should now be able to answer:

• How do you calculate the per-second probe rate for a synthetic suite at a given cadence?
• Which budget is the cheapest to exhaust: exporter CPU, scrape concurrency, TSDB head series, or target load?
• When is an external vantage point worth the egress bill, and when is an internal vantage point sufficient?
• Why is the journey’s cadence measured in minutes rather than seconds?
• What is the inventory discipline that keeps a synthetic suite right-sized over time?

Quiz