blackbox_exporter Overview

At 03:12 the on-call engineer opens a panel: probe_success for twelve external services. Eleven are green. One is red. The service inside the red row is up by every internal metric the team has, but a customer in Frankfurt cannot reach it. The single panel that resolves the ambiguity is the one driven by blackbox_exporter. This lesson is the foundation that makes that panel exist.

What it is

blackbox_exporter is a Prometheus exporter that turns external protocol probes into the standard Prometheus exposition format. It exposes a single HTTP endpoint, conventionally /probe, that accepts a target parameter and a module parameter, performs the probe, and returns one sample per attempt in the same form any other exporter would. Prometheus then scrapes that endpoint; the act of scraping is the probe. The exporter does not pull targets of its own and does not schedule work; it is queried.

The canonical alternative is smokeping or a generic synthetic service (Pingdom, Grafana Synthetic Monitoring). The exporter is the right answer when the team already runs Prometheus, because the metric shape matches everything else in the stack. A pure synthetic service is the right answer when the probe must run from many geographies or from inside a managed browser; the exporter covers the per-region core probes that run from a host or container you control.

Why a sysadmin cares

Three production failure classes disappear the day the exporter is in place:

• Route blackouts. The service is up internally; the user-facing URL is down because of a load balancer, a reverse proxy, a CDN edge, or a DNS record. Internal white-box metrics cannot see this.
• External dependency failure. A managed Postgres, an S3-compatible object store, a third-party API. The application is up. The dependency is not reachable from the production path. Internal metrics describe only the application.
• Silent configuration drift. A 503 that only manifests for one percent of users because a default backend fell out of a load-balancer pool at 02:00. The probe catches it within seconds because the probe path is the same path the user takes.

The exporter is a load-bearing component of the observability platform. It is also small, runs in a single container, and costs almost nothing in CPU. The reason to skip it is “we have internal monitoring”; the cost of skipping it is the next incident where the user is right and the internal panel is wrong.

How it works

The exporter exposes one HTTP handler, /probe. Prometheus sends a request with two query parameters: target, the endpoint to probe, and module, the named probe configuration from the exporter’s config file. The exporter executes the probe, collects per-protocol metrics, and returns them in Prometheus exposition format. There is exactly one scrape per target; the exporter does not multiplex internally.

   prometheus.yml
        |
        v
  scrape job: blackbox_http
        |
        | each scrape rewrites the target into
        | the static_config on the exporter
        |
        v
   blackbox_exporter  :9115/probe?target=X&module=Y
        |  ----- module: http_2xx / tcp_connect / icmp / dns ----
        |
        | probe runs, measurements returned as Prometheus metrics
        |
        v
   probe_success, probe_duration_seconds,
   probe_http_status_code, probe_ssl_earliest_cert_expiry,
   ...

The scrape job drives the cadence. The exporter drives the protocol details. Prometheus stores the result alongside every other metric in the platform. Grafana and Alertmanager consume it through the same query and routing pipelines as any other metric.

The module catalogue

A module is a named probe configuration. The exporter ships a small set of first-party modules; each module is a vocabulary for one protocol. The first-party modules and their questions:

• http_2xx — open a TCP connection, perform a TLS handshake if the URL scheme is https, send an HTTP request, treat any response in the 2xx range (by default) as success. Surfaces probe_http_status_code, probe_http_content_length, probe_http_redirects, probe_ssl_earliest_cert_expiry.
• http_4xx — same probe, but success is defined as a 4xx status code. Useful for catching “expected absence” — for example, a page that must return 404.
• http_xxx — a custom variable: success when the response is in the listed set (e.g., http_2xx succeeds on [200, 204]). Useful when the application has a single canonical success status that is not the default.
• tcp_connect — open a TCP connection to target:port. No application handshake. Success is a successful connect.
• icmp — issue an ICMP echo request. Success is any reply within the timeout. Requires CAP_NET_RAW or root.
• dns — issue a DNS query for a configured name and type. Validate the answer against regexes and authority sections.
• ssl — perform a TLS handshake without an HTTP request. Surfaces probe_ssl_earliest_cert_expiry without an application probe. The lesson on TLS probes covers this.
• irc, pop3s, sip, ssh_proxyprotocol — also shipped, used in narrow contexts. Most production deployments use only the six above.

  module          | protocol question asked
  ----------------+------------------------------------
  http_2xx        | does the URL return success?
  http_4xx        | does the URL return the expected 4xx?
  tcp_connect     | is a TCP port reachable?
  icmp            | does the host reply to ping?
  dns             | does the resolver return the expected answer?
  ssl             | is the certificate chain valid and fresh?

Each module exposes the same outer metrics (probe_success, probe_duration_seconds) plus protocol-specific ones. The shape is the same across modules, which is what makes dashboards and alerts reusable.

The metric shape

The exporter does not invent a new metric namespace per module. The convention is fixed and stable across versions:

• probe_success — gauge, value 1 on success and 0 on failure. This is the headline metric.
• probe_duration_seconds — gauge, total wall-clock duration of the probe including DNS resolution, connect, TLS, and any redirects. Useful for latency drift.
• probe_failed_due_to_regex — gauge, 1 if the probe’s response-body or DNS-answer regex did not match, 0 otherwise.
• probe_http_status_code — gauge, the final HTTP status code observed.
• probe_http_redirects — counter, the number of HTTP redirects followed.
• probe_ssl_earliest_cert_expiry — gauge, Unix timestamp of the earliest certificate expiry across the chain.
• probe_dns_lookup_time_seconds — gauge, time spent in DNS resolution.

All of these carry the labels Prometheus attaches during the scrape (instance, job) and any labels the user added via relabeling. The exporter does not label by itself; module name goes on as a label through relabeling, never on the wire.

How to configure it

Three layers are required to put a probe into production: the exporter’s own config, the scrape job on Prometheus, and (optionally) the relabeling layer that maps each probe target to a high-cardinality, low-cardinality label set.

1. The exporter config: blackbox.yml

# /etc/blackbox/blackbox.yml
modules:

  # HTTP success probe used for the public checkout flow.
  # tls_config controls the verifier, not the application TLS.
  http_2xx_checkout:
    prober: http
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4       # try IPv4 first
      ip_protocol_fallback: true       # fall back to IPv6
      follow_redirects: true           # honour 3xx up to 10 hops
      fail_if_body_matches_regexp:
        - "checkout disabled"
      fail_if_header_matches:          # refuses to serve when ...
        - header: content-type
          allow: false
          regexp: "text/html"
        - header: cache-control
          allow: false
          regexp: "no-store"
      tls_config:
        insecure_skip_verify: false    # validate the chain

  # TCP connect for database-port reachability (see lesson on TCP).
  tcp_connect_pg:
    prober: tcp
    timeout: 3s

  # ICMP for host liveness (see lesson on ICMP).
  icmp_router:
    prober: icmp
    timeout: 2s
    icmp:
      preferred_ip_protocol: ip4

  # DNS probe for the public resolver (see lesson on DNS).
  dns_a_shop:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      query_name: shop.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp: []
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.[0-9]{1,3}\\.[0-9]{1,3})$"

  # SSL handshake for certificate-only monitoring (see lesson on TLS).
  ssl_shop:
    prober: http                  # SSL probes ride the http prober
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      tls_config:
        insecure_skip_verify: false

  # An http_xxx custom success class.
  http_health_200_204:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200, 204]
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

Each module is independent. A module can be reused by many scrape jobs; a target can be probed by many modules. The naming discipline matters: name the module after the question, not the protocol. dns_a_shop and dns_aaaa_shop are clearer than dns1 and dns2.

2. The Prometheus scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_http
    metrics_path: /probe
    params:
      module: [http_2xx_checkout]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - https://shop.example.com/checkout
        labels:
          service: shop
          env: prod
          region: eu-west-1
    relabel_configs:
      # The exporter needs the real target on every scrape URL.
      - source_labels: [__address__]
        target_label: __param_target
      # Pick up the service label from the static_config.
      - source_labels: [service]
        target_label: service
      # Stamp the module name onto every sample as a label.
      - target_label: module
        replacement: http_2xx_checkout
      # Keep instance set to the prober, not the target.
      - source_labels: [__address__]
        target_label: __tmp_target_address
      - source_labels: [__tmp_target_address]
        regex: '(.*):(.*)'
        replacement: '${1}'
        target_label: instance

The pattern to internalise: __param_target is rewritten from __address__ so the exporter knows what to probe. The exporter instance itself (localhost:9115) is what Prometheus scrapes; the instance label is normally rewritten to the real target so that alerts group by service, not by prober host.

3. Validation of the exporter config

# The exporter has no built-in check; the binary refuses to start
# with an invalid config, which is itself a check.
blackbox_exporter --config.check --config.file=/etc/blackbox/blackbox.yml
journalctl -u blackbox_exporter --since '5 min ago' | tail -20

# Validate the Prometheus side.
promtool check config /etc/prometheus/prometheus.yml

How to validate it

Three layers must be confirmed: the exporter is up, the scrape is flowing, and the probe actually exercises the path you think it does.

# 1. The exporter is up. Reads /metrics from the exporter itself,
#    not from the probe.
curl -sf http://localhost:9115/metrics | grep -E '^blackbox_'
# ... illustrative ...
# blackbox_exporter_build_info{version="0.26.0"} 1
# blackbox_exporter_config_last_reload_successful 1
# blackbox_exporter_probe_total 18

# 2. Smoke-test the probe manually before trusting Prometheus.
curl -sf "http://localhost:9115/probe?module=http_2xx_checkout&target=https://shop.example.com/checkout" \
  | grep -E '^(probe_success|probe_duration_seconds|probe_http_status_code)'
# probe_duration_seconds 0.214
# probe_http_status_code 200
# probe_success 1

# 3. Confirm Prometheus is receiving the series.
up{job="blackbox_http"}
# {instance="shop.example.com", job="blackbox_http", service="shop"} 1

probe_success{service="shop"}
# 1

# 4. The /-/ready endpoint of the exporter reports modules loaded.
curl -sf http://localhost:9115/-/ready
# ready

If probe_success is 1 and probe_http_status_code is 200, the probe is exercising the right path. If probe_success is 0, inspect probe_failed_due_to_regex and the exporter stderr log for the underlying reason.

How it can fail

The probe is the simplest component of the platform, and its failure modes are predictable.

1. Wrong target URL. __param_target is missing the scheme or carries the exporter’s address instead of the real target. Every probe returns probe_success=1 against localhost, dashboards stay green, the route outage continues. Symptom: the instance label is blackbox:9115 not shop.example.com.

2. Module returns success against the wrong question. The module expects JSON but the application returns HTML for a 200; the regex matches accidentally on every page. The probe returns green; the user-facing flow is broken. Symptom: probe_failed_due_to_regex is 0 but the application SLO is breached.

3. Module timeout shorter than the slowest legitimate response. SRE introduces a new dependency that adds 800 ms on cold cache. The probe fires for: 1m before the dependency responds. Symptom: probe_success=0 with probe_duration_seconds close to the timeout value.

4. ICMP blocked upstream. Many cloud providers and most corporate firewalls drop ICMP beyond the perimeter. The probe gets probe_success=0 for healthy hosts. Symptom: probe_success{module="icmp_router"}=0 while every application-level probe is green.

5. Relabeling rewrites break cardinality. A relabel rule copies the entire URL into a label. Five hundred targets times the unique path component produces a series storm. Symptom: Prometheus memory rises, TSDB head block churns.

6. The exporter host’s own egress is broken. Every probe fails because the exporter cannot reach the public internet. Symptom: all service labels red at once; the pattern is every target, not one target. This is the signal that the problem is local.

7. Module drift on upgrade. A 0.x version renames a regex match key. The old key silently no-ops. The probe keeps returning green. Symptom: behaviour matches an older major release even after a restart with a new binary.

How to troubleshoot it

Order matters. Start at the boundary where you have evidence.

1. Is the exporter alive? curl http://exporter:9115/-/ready and curl http://exporter:9115/metrics first. A missing blackbox_exporter_build_info series means the exporter is not the version you think, or it died. Look at systemd / container logs.
2. Does Prometheus see the exporter? up{job="blackbox_*"} in PromQL. If up=0, the problem is at the scrape boundary, not at the target.
3. Does the probe itself work? Re-run the exact URL the Prometheus scrape would produce, against the exporter manually: curl "http://exporter:9115/probe?module=...&target=...". The headers and body tell you which boundary failed.
4. Compare probe paths to internal paths. When a probe reports failure but up{job="node"} for the internal service is green, you have isolated the failure to the external path.
5. Check probe_duration_seconds. A drift from 200 ms to 4 s while probe_success=1 is a brownout that has not yet failed; it precedes the failure.
6. Check the exporter host’s own egress. A curl from the exporter host to the target is the fastest test.
7. Compare modules against each other. If tcp_connect_pg is green and http_2xx_checkout is red, the failure is in the application path, not the network. If both are red on one target only, the target is the problem.

Security implications

The exporter accepts arbitrary target query parameters when exposed without auth. That single endpoint, unauthenticated, can be coerced into probing any IP or port the exporter host can reach. In a multi-tenant environment that becomes a port-scan primitive. Bind the exporter’s /probe to a network Prometheus can reach but untrusted users cannot, or run it on a private network.

tls_config.insecure_skip_verify: true makes the probe accept a TLS handshake against an expired, self-signed, or wrong-host certificate. A single misconfigured module means probe_success=1 while every TLS-using user is rejected. The dashboard is green; the platform is broken. Never ship a production module with insecure_skip_verify: true unless the lesson’s TLS section has been read and the trade-off accepted.

The metric labels are produced by relabeling. A relabel rule that copies a user-controlled URL fragment into a label is a cardinality attack. Whitelist the labels.

Performance implications

The exporter is cheap per probe. The cost scales as the cardinality of the instance and service label sets, the scrape interval, and the time budget of the slowest legitimate response.

• Probe interval. Default 30 s is reasonable. Faster intervals (5 s, 10 s) improve time-to-alert on outages but raise scrape load proportionally.
• Timeout. Each timeout is a goroutine held in the exporter until the deadline. A misconfigured timeout: 60s on a hundred-target scrape job can occupy the exporter for the entire scrape interval.
• Cardinality. instance=host plus service=... plus module=... is the recommended minimum. Anything more, like full URLs in labels, is a budget problem.
• Timeouts and the scrape budget. scrape_timeout on Prometheus must be longer than any module’s timeout. A module with timeout: 5s scraped every 30 s with scrape_timeout: 5s races against the exporter.

Production guidance

• Treat the blackbox.yml and the scrape job as production code. Both go through review and CI.
• Pin the module names that alerts and dashboards reference. Never rename a module without a coordinated migration.
• Run at least one probe from outside the application’s network (a separate exporter host in a different VPC or region) and one from inside (so the user-path questions differ).
• Validate the exporter config with --config.check. Validate the Prometheus side with promtool check config. Both must pass before the change ships.
• Add a single SLO: probe_success == 1 for each service label. Alert when it is 0 for two scrapes in a row. Do not import the metric into a SLO with a higher bar than the probe can meet.

Verification

You should now be able to answer:

• What problem does blackbox_exporter solve that white-box metrics do not?
• Which module answers each of these questions: “is the checkout page reachable from outside?”, “is TCP port 5432 open?”, “does the host reply to ping?”, “does DNS return the expected A record?”
• How is __param_target wired from a Prometheus scrape job to the exporter’s /probe handler?
• Which four boundaries can a blackbox probe failure live at, and which metric reveals each?
• What is the operational cost of running the probe path with insecure_skip_verify: true?

Quiz