Synthetic Monitoring Overview

A user opens a ticket. The product page loads fine from the office, fine from the engineer’s laptop on a coffee-shop network, and fine from a smoke-test in staging. The customer is in a different region, on a residential ISP, behind a CGNAT, and the page loads as a blank white rectangle. The dashboard is green. The application logs show no error. The CDN panel shows a healthy cache-hit ratio for everyone else. The customer’s report sits in the queue for ninety minutes before someone starts to believe them.

Synthetic monitoring is the discipline of asking the system questions from outside, on a schedule, before a user has to. It is the complement to in-app telemetry, not a replacement.

What it is

Synthetic monitoring is the practice of running a fixed probe against a known target, on a fixed cadence, from a fixed (or small set of) locations, and recording the outcome as a metric. The probe is synthetic because it stands in for a real user without being one. The result is a time series of probe_success and probe_duration_seconds that the platform can chart, alert on, and post-mortem with.

The canonical Prometheus implementation is blackbox_exporter (Prometheus 2.55.x targets blackbox_exporter 0.26.x). The exporter is a single static binary; it speaks the Prometheus exposition format over an HTTP endpoint and accepts a target plus a module name as URL parameters. The exporter does not store anything. Prometheus scrapes it; rules and alerts run on the result.

The other half of synthetic monitoring is the scripted journey: a multi-step probe that drives a real browser or HTTP client through a user flow such as login → search → checkout. In the Prometheus ecosystem this is usually run by a separate scripted prober (k6, Playwright, Grafana Cloud k6, internal microservice) whose results are pushed to a Pushgateway or to a remote-write receiver.

Why a sysadmin cares

Synthetic monitoring is the only signal class that asks “what does the system look like from the outside, right now” without needing a user to be present. Three production problems map cleanly onto it:

• Cold path detection. An application that has not been hit in hours is cold. The first request after idle pays the JIT, the connection-pool warm-up, the cache fill. Synthetic probes hit the cold path on a schedule and the dashboard shows the cold-start tax before the user does.
• Boundary detection. A bad deploy that breaks the external interface (TLS handshake, redirect chain, CORS preflight) while leaving the internal metrics green. The internal metrics watch the inside; the synthetic probe watches the boundary.
• Region and provider detection. A failure that is visible only from one ASN, one region, or one provider. No single in-app metric detects this because the metric sees only its own requests; the synthetic probe is the cheap way to add a vantage point.

The probe is also the cheapest way to assert an SLO before a release. “Checkout returns 200 within 800 ms from three regions” is a question only synthetic monitoring answers reliably, and it can answer it on every canary.

How it works

The mental model has three moving parts: the probe source, the exporter, and the scrape.

   probe source               exporter                Prometheus
   ------------               --------                ----------
   scheduled cron -->  blackbox_exporter  <--scrape--  prometheus.yml
        |                       |                       |
        |   GET /probe?         |                       |
        |   module=http_2xx     |                       |
        |   &target=...:443     |                       |
        |                       |                       |
        |                       |---- probe_* ---->     |
        |                       |                       |
        |                       |     rules + alerts    |
        v                       v                       v
    one-off test          continuous probe        time-series store

The exporter is stateless. It does not decide whether to probe; it answers what the probe said this time. Prometheus decides the cadence via scrape_interval and the set of targets via static_configs, file_sd_configs, or service discovery.

There are two distinct operating modes:

• Pull mode. Prometheus scrapes /probe on the exporter once per scrape_interval for each target. The exporter is treated as a regular scrape target. The metric flow is exporter -> Prometheus. This is the default for blackbox_exporter and is what every lesson in this module assumes.
• Push mode. A scripted prober runs on a schedule and pushes its results to a Pushgateway, which Prometheus scrapes. This is the right shape for browser-driven multi-step journeys, because the prober owns the timing and the user-agent identity; Prometheus cannot easily pull from a browser.

The two modes are complementary. Pull-mode probes are inexpensive and run at high frequency. Push-mode journeys are expensive and run at low frequency. The right design uses both.

Synthetic versus real-user monitoring

The distinction is the source of the telemetry, and the distinction matters.

  synthetic probe            real-user monitoring (RUM)
  ---------------            -------------------------
  fixed vantage point        arbitrary user locations
  fixed cadence              event-driven, by user action
  fixed user-agent           arbitrary user-agents
  known payload              arbitrary payloads
  no business state          carries user session, basket,
                             auth token, account state

Synthetic is pre-defined. It watches for the failure shapes you remembered to script. RUM is post-hoc. It watches for the failure shapes that real users encountered, including the ones you did not anticipate.

A common mistake is to treat synthetic as a substitute for RUM. It is not. A synthetic probe that hits /healthz tells you the health endpoint is up. It does not tell you the user with a three-year-old Android, an old TLS stack, and a regional routing problem is unable to complete checkout. RUM is the only telemetry class that captures that signal.

The right mix in production is:

• In-app telemetry — RED metrics (rate, errors, duration), USE metrics (utilisation, saturation, errors), structured logs with correlation IDs, traces for the slow path. This is the ground truth. The application knows what the user did.
• Synthetic probes — HTTP 2xx on the public boundary, TLS handshake and chain, DNS resolution, multi-step journey from two or three vantage points. This is the boundary assertion. The synthetic probe knows what a user should be able to do.
• RUM — when budget allows, a small sampled stream from real browsers, capturing the failures the in-app and synthetic signals miss.

The three classes are not interchangeable. The synthetic probe is the earliest warning because it runs on a schedule and does not need a user to be present.

How to configure it

The minimum viable deployment is a blackbox_exporter process and a Prometheus scrape job. Below is a production shape with one of each module type represented.

# /etc/blackbox/blackbox.yml
modules:

  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: true

  tcp_connect_pg:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_http
    metrics_path: /probe
    params:
      module: [http_2xx_example]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - https://example.com/healthz
          - https://api.example.com/v1/ping
        labels:
          team: platform
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(.*)'
        replacement: '${1}'
        target_label: target
      - target_label: __address__
        replacement: blackbox.internal:9115

The two relabel lines do the real work: __param_target becomes the URL parameter, and __address__ becomes the exporter’s address. Without the relabel, Prometheus scrapes the exporter with target= empty and the probe fails by configuration.

For scripted journeys, the exporter is not the right tool. The right shape is a small service that runs the journey on a cron, computes a single probe_success per journey step, and pushes:

# /etc/prometheus/prometheus.yml (Pushgateway)
scrape_configs:
  - job_name: pushgateway_synthetic_journey
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

The journey runner is out of scope for blackbox_exporter but it is the right shape for any flow that needs a real browser, a session cookie, or a payment token.

How to validate it

# 1. Confirm the exporter is alive.
curl -sf http://blackbox.internal:9115/-/healthy
# Prometheus Blackbox Exporter is Healthy.

# 2. Run one probe by hand.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_example' \
  --data-urlencode 'target=https://example.com/healthz' \
  | grep -E '^probe_'
# probe_duration_seconds 0.083
# probe_failed_due_to_regex 0
# probe_http_status_code 200
# probe_http_version 1.1
# probe_ip_protocol 4
# probe_ssl_earliest_cert_expiry 1.765e+09
# probe_success 1

# 3. Confirm Prometheus has the metric.
probe_success{team="platform",env="prod"}
# {target="https://example.com/healthz"} 1

# 4. Validate the blackbox.yml is parseable.
docker run --rm -v /etc/blackbox:/config prom/blackbox-exporter:v0.26.0 \
  --config.check --config.file=/config/blackbox.yml
# (no output means OK)

# 5. Validate Prometheus scrapes parse.
promtool check config /etc/prometheus/prometheus.yml
# SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config

A green exporter health endpoint, a green probe_success, and a green promtool are the three signals the deployment is wired correctly. The next step is to leave it for one hour and confirm the time series actually accumulates.

How it can fail

1. Single vantage point. The probe runs from inside the cluster, or from one cloud region, or from one ASN. A regional DNS failure, a provider-level routing problem, or a CDN POP outage is invisible. Symptom: production is down, the synthetic probe is green, the alert never fires.

2. Scrape budget exhaustion. Blackbox jobs have a reputation for “small, cheap, throw them everywhere.” The reputation is wrong. A scrape job with 1 000 targets at scrape_interval=15s is a 67 rps load on the exporter and 67 rps of network egress. The exporter exhausts its file descriptors; Prometheus logs context deadline exceeded. Symptom: probes return intermittently, the dashboard has gaps, the alert is missing data.

3. Relabel bug copies target into instance naively. The instance label is set to the full URL. Every target becomes a new series. Cardinality rises with the target list. Symptom: TSDB head series climbs unaccountably; the metric’s instance label is unusable in alerts.

4. Module mismatch. The scrape job asks for module http_2xx but the exporter has http_2xx_example. The exporter returns an error and probe_success=0 for every probe in the job. Symptom: every probe in the job red, the rest of the exporter green.

5. preferred_ip_protocol without fallback. The hostname resolves to IPv6 only; the exporter is configured with preferred_ip_protocol: ip4 and ip_protocol_fallback: false. Every probe fails by configuration. Symptom: every probe red, the application is up, the dashboard does not agree with reality.

6. The probe is testing the wrong thing. The probe hits a static /healthz that returns 200 even when the application is wedged. The probe is green; the application is broken. Symptom: probe green, application broken, no alert.

7. Probes from inside the trust zone. The probe runs from a host in the same VPC as the application, bypassing the load balancer and the WAF. The probe is green while the public surface is broken. Symptom: probe green, customer reports red, on-call asks why.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

1. Is the exporter alive? curl http://exporter:9115/-/healthy. If this fails, the exporter process is the boundary.
2. Does the module exist? curl http://exporter:9115/probe?module=NAME&target=.... The exporter returns Error: No such module for typos.
3. Does the target resolve from the exporter host? getent hosts example.com or dig example.com @exporter-host from the same network. A DNS problem looks like a probe problem until you check.
4. Is the target reachable from the exporter host on the target port? nc -vz example.com 443. If this fails, the network is the boundary.
5. Run the probe by hand and read the full output. Look at probe_failed_due_to_regex, probe_failed_due_to_tls, probe_http_status_code. The exporter tells you why a probe failed; do not reason from probe_success=0 alone.
6. Confirm Prometheus is scraping it. up{job="blackbox_http"} must be 1. If not, the relabel is the boundary.
7. Confirm the label set is sane. Run count by (__name__) ({__name__=~"probe_.*"}). Cardinality that does not match the target count is a relabel bug.

Security implications

The exporter accepts an arbitrary target as a URL parameter. Without care, the exporter becomes a port-scan primitive — an attacker who can reach the exporter can ask it to probe any IP and port the exporter host can reach. Bind the exporter to a private network. Do not expose it on the public internet without an authenticating reverse proxy.

For scripted journeys, the journey runner holds credentials (session cookie, API token, payment instrument). Those credentials must be rotated regularly and stored in a secret manager, not in the prober’s environment file. The prober’s logs must redact the credentials; the platform’s logs must not contain the prober’s environment.

TLS probes validate the certificate chain. A misconfigured TLS probe with insecure_skip_verify: true records probe_success=1 against any certificate. The audit will read the production config. Do not bypass the chain.

Performance implications

A synthetic probe is a load. The cost is:

• Exporter CPU. Each probe allocates goroutines, runs the configured prober, renders the metric set. A modern four-core exporter handles roughly 200 probes per second on the HTTP module before saturating.
• Network egress. Each probe is a request from the exporter host to the target. 1 000 targets at 30 s intervals is ~33 rps outbound from the exporter. Pin this against the egress budget.
• Prometheus storage. probe_* metrics are eight to twelve series per target. A 1 000-target suite is roughly 10 000 active series in the TSDB head.
• Scrape budget. Each scrape job occupies a slot in Prometheus’s global concurrency limit. Synthetic suites are usually the first to exhaust it.

The right mitigation is right-sized. A 200-target suite at 60 s intervals is 3.3 rps, well within budget. A 1 000-target suite at 15 s intervals is 67 rps and will saturate the exporter.

Production guidance

• Pick vantage points deliberately. At least two regions, and one external vantage point if the budget allows. Synthetic from inside the trust zone is a partial answer.
• Pick cadence deliberately. 60 s is the default for blackbox. 15 s is justified for the public boundary. Sub-minute is justified for the payment path. Avoid sub-second cadences — they fight the scrape budget for no operational benefit.
• Treat the probe target as code. The URL, the headers, the module name all live in version control.
• Set scrape_timeout to 2 * scrape_interval and the module timeout to less than the scrape timeout. Otherwise, the probe exceeds the scrape window and Prometheus marks it failed before the exporter answers.
• Validate the config with blackbox_exporter --config.check before every reload.
• Relabel to strip the path from instance. Group alerts by service, not by URL.
• Run a small suite, not a comprehensive one. Ten probes per service is usually enough to bound the failure surface.

Verification

You should now be able to answer:

• What does synthetic monitoring answer that in-app telemetry does not, and what does it not answer that in-app telemetry does?
• Where does blackbox_exporter sit in a Prometheus 2.55.x stack, and what is its scope?
• When is push-mode synthetic (a Pushgateway job) the right shape instead of a pull-mode blackbox probe?
• Why is a single vantage point a structural failure of the synthetic design?
• What is the scrape budget cost of a 1 000-target synthetic suite at 15 s intervals?

Quiz