HTTP Probes

The HTTP probe returns green. The dashboard says the public boundary is healthy. The customer opens a ticket, screenshots a half-loaded checkout page, and writes “this has been broken since Tuesday.” The on-call opens the dashboard. The probe is green. The screenshot shows a 200 response that contains the HTML shell but not the application data — the page returned 200 because the shell returned 200; the embedded JavaScript failed silently because the API behind it returned 503, and the user sees a spinner that never resolves.

The HTTP probe, on its own, never claimed to be a user-flow probe. The lesson is about what http_2xx actually does and the operational discipline of choosing the path and the status code deliberately.

What it is

The http_2xx module of blackbox_exporter 0.26.x performs a single HTTP request against the configured target and validates the response against a set of rules. The request is issued from the exporter host, by the exporter’s Go HTTP client, with the configured method, headers, TLS configuration, and timeout. The validation is the configurable part: which HTTP versions are acceptable, which status codes are acceptable, whether the response body must match a regex, whether redirects are followed.

The headline metric is probe_success. The HTTP-specific metrics are probe_http_status_code, probe_http_version, probe_http_duration_seconds (the time spent waiting for the response), probe_failed_due_to_regex, probe_failed_due_to_tls, and probe_redirects (the number of hops followed before the terminal response).

The module does not execute JavaScript. It does not render HTML. It does not interpret the response. A green probe means the configured rules passed for this single request. It does not mean the user flow works.

Why a sysadmin cares

The HTTP probe is the workhorse of the synthetic suite. It is the cheapest signal that the public boundary is up, that TLS terminates, that the load balancer answers, and that the application returns a meaningful response. Three production questions map onto it:

• Is the public boundary up? A probe against https://example.com/healthz is the cheapest assertion.
• Is the redirect chain healthy? A probe with follow_redirects: true validates the chain end-to-end, including the final hop.
• Is the API surface answering correctly? A probe with valid_status_codes: [200] against the canonical API endpoint validates the response shape the user gets.

The probe is also the right shape for a canary gate. “Block the rollout if the boundary probe is red for two consecutive intervals” is a single rule.

How it works

The exporter’s HTTP prober is a wrapper around Go’s net/http client. The prober resolves the target, opens a connection (HTTP or HTTPS, depending on scheme), sends the configured request, reads the response, applies the validation rules, and records the result.

  exporter host                                 target host
        |                                             |
        | --- TCP SYN --->                            |
        | <-- TCP SYN-ACK ---                         |
        | --- TLS ClientHello -->                     |
        | <-- TLS ServerHello, cert, done --          |
        | --- HTTP request -->                        |
        |                                             |
        | <-- HTTP response (status, headers, body) - |
        |                                             |
        | --- TLS close_notify -->                    |
        | --- TCP FIN -->                            |
        v                                             v
  apply rules:                                      served
   - valid_http_versions                              |
   - valid_status_codes                               |
   - body regex match                                 |
   - fail_if_ssl / fail_if_not_ssl                    |
   - follow_redirects (if any)                        |
        |
        v
  probe_success = 1 or 0
  probe_http_status_code = <code>
  probe_http_version = <1.1 or 2.0>

Two configuration choices dominate the result: valid_status_codes and valid_http_versions. The default behaviour is “any 2xx” — that is too loose for production. A disciplined probe names the exact set of acceptable codes.

How to configure it

Below is a production-shaped blackbox.yml with four HTTP variants a typical environment needs.

# /etc/blackbox/blackbox.yml
modules:

  # Cheap public boundary probe.
  # Returns 200, must be HTTPS, must not redirect.
  http_2xx_boundary:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: true
      headers:
        User-Agent: blackbox-exporter/0.26

  # Login-page probe that follows the redirect chain.
  # Validates that the redirect from /login -> /auth works
  # and the final hop returns 200.
  http_2xx_login:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      max_redirects: 5
      fail_if_not_ssl: true

  # API probe that POSTs a payload and validates the body
  # shape with a regex.
  http_2xx_api:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 204]
      method: POST
      headers:
        Content-Type: application/json
      body: '{"probe": true}'
      fail_if_matches_regexp:
        - '"status":"error"'
      fail_if_not_matches_regexp:
        - '"status":"ok"'

  # Strict probe that requires HTTP/2 and rejects the body
  # if it matches an error signature.
  http_2xx_strict:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      fail_if_matches_regexp:
        - 'service unavailable'
        - 'maintenance'
      fail_if_not_matches_regexp:
        - 'OK'

The scrape job in Prometheus ties the module to the targets.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_http_boundary
    metrics_path: /probe
    params:
      module: [http_2xx_boundary]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - https://example.com/healthz
          - https://api.example.com/v1/ping
        labels:
          service: public-boundary
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(https?://[^/]+)(/.*)?'
        replacement: '${1}'
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

The status code, the HTTP version, the path, and the body match are the four levers the operator has. Each one should be set deliberately.

How to validate it

# 1. The probe against the real target.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_boundary' \
  --data-urlencode 'target=https://example.com/healthz' \
  | grep -E '^probe_'
# probe_duration_seconds 0.082
# probe_failed_due_to_regex 0
# probe_http_duration_seconds 0.061
# probe_http_status_code 200
# probe_http_version 1.1
# probe_ip_protocol 4
# probe_ssl_earliest_cert_expiry 1.765e+09
# probe_success 1

# 2. The same probe against a target that should redirect.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_boundary' \
  --data-urlencode 'target=http://example.com/healthz' \
  | grep -E '^probe_'
# probe_failed_due_to_regex 0
# probe_http_status_code 301
# probe_success 0     # because fail_if_not_ssl: true

# 3. The body-match probe against a healthy and broken target.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_api' \
  --data-urlencode 'target=https://api.example.com/v1/check' \
  | grep -E '^probe_failed_due_to_regex|^probe_success'
# probe_failed_due_to_regex 0
# probe_success 1

# 4. Confirm Prometheus has the metric.
probe_http_status_code{service="public-boundary",env="prod"}
# {target="https://example.com/healthz"} 200

# 5. Validate the exporter config.
docker run --rm -v /etc/blackbox:/config prom/blackbox-exporter:v0.26.0 \
  --config.check --config.file=/config/blackbox.yml
# (no output means OK)

A green probe_success plus probe_http_status_code matching the expected status is the headline assertion. fail_if_ssl, fail_if_not_ssl, and the body regex are the secondary assertions.

How it can fail

1. Path returns 200 but the application is broken. A /healthz endpoint that the operator wired to return 200 unconditionally. The probe is green; the application is wedged. Symptom: probe green, application logs full of 5xx, user reports mounting.

2. valid_status_codes is too loose. The default is any 2xx. The application returns 203 Non-Authoritative Information on a broken path; the probe accepts it. Symptom: probe green, the user-visible flow returns “wrong content.”

3. follow_redirects: false on a path that always redirects. The probe records the redirect; the probe fails because the redirect is not in valid_status_codes. Symptom: every probe red, the redirect is intentional.

4. fail_if_not_ssl: false on a probe that must be HTTPS. The probe silently accepts a downgrade to HTTP. Symptom: probe green, MITM possible in theory, no alert.

5. Body regex matches too much. A regex that matches the whole HTML page. The probe never fails on the body. Symptom: probe green regardless of body content.

6. Timeout shorter than the cold path. A probe against a cold-start application with a 10-second cold cache and a 5-second timeout. Probe fails every cold probe. Symptom: probe flaps, latency panel shows timeouts, the dashboard matches the user report.

7. Scrape interval shorter than probe duration. A 30 s scrape interval against a target with a 25 s probe duration. The next scrape starts before the previous one ends. Symptom: probe returns context deadline exceeded, probe_success=0, exporter logs saturation warnings.

8. Method mismatch. The probe uses method: GET against an endpoint that requires POST. The server returns 405. Symptom: probe red, the endpoint is the wrong one.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

1. Run the same request by hand. curl -sv https://example.com/healthz. If this fails, the network or the application is the boundary; the probe was honest.
2. Run the probe by hand and read the full output. probe_failed_due_to_regex, probe_failed_due_to_tls, probe_http_status_code, probe_http_version. The exporter tells you why a probe failed; do not reason from probe_success=0 alone.
3. Confirm the status code matches the configured set. A 301 from a target that must return 200 is a configuration choice on the server side, not a probe failure.
4. Confirm the TLS chain if fail_if_ssl or fail_if_not_ssl is set. A red probe with probe_failed_due_to_tls=1 is a TLS problem; do not blame the HTTP layer.
5. Confirm the body regex matches the actual response. curl https://target | grep -E '<regex>'. A regex that no longer matches after a deploy is a real failure; the application changed shape.
6. Compare probe latency to baseline. A drift from 100 ms to 4 s on an HTTP probe is a brownout; the next failure is a timeout.
7. Confirm scrape_timeout exceeds the longest expected probe duration. Otherwise the scrape window closes before the probe returns.

Security implications

The exporter accepts an arbitrary target as a URL parameter. The same port-scan risk applies; the HTTP module lets an attacker who can reach the exporter exercise any URL the exporter host can reach. Bind the exporter to a private network.

The HTTP module also lets the operator send arbitrary headers and bodies. Do not embed credentials in headers; the exporter logs the request and the body lives in Prometheus’s scrape logs. For endpoints that require authentication, prefer the canonical authentication probe in the application layer; the blackbox probe is for boundary assertions, not authenticated flows.

fail_if_not_ssl: true is a meaningful assertion. It fails the probe if the server responds in cleartext. A production probe of an HTTPS target that does not include this assertion accepts a silent downgrade. The audit will read the production config.

Performance implications

An HTTP probe is more expensive than a TCP probe: it performs the TCP handshake, the TLS handshake, the HTTP request, and the response read. The cost is roughly:

• Exporter CPU. Each probe runs a goroutine and allocates an http.Client. A modern four-core exporter handles roughly 100 HTTP probes per second before saturating.
• Network egress. Each probe is a request from the exporter host to the target. 1 000 targets at 30 s is 33 rps outbound.
• TLS handshake cost. The TLS handshake is the most expensive part of an HTTPS probe. Session resumption mitigates this for repeated probes to the same target; the exporter does not currently cache TLS sessions.

The right mitigation is right-sized. A 200-target suite at 60 s intervals is 3.3 rps, well within budget. A 1 000-target suite at 15 s intervals is 67 rps and will saturate the exporter and exhaust the scrape budget.

Production guidance

• Pick valid_status_codes deliberately. The default “any 2xx” is too loose for a production SLO. The probe should name the exact set of acceptable codes.
• Pick the path deliberately. /healthz is fine for the cheapest assertion, but it does not prove the user flow works. Promote to a multi-step journey when the question is richer than “the boundary is up.”
• Set fail_if_not_ssl: true for any probe against an HTTPS target. The assertion catches silent downgrade.
• Set fail_if_ssl: false for HTTPS targets; true is for the inverse, a probe that must hit cleartext.
• Set follow_redirects: true when the path intentionally redirects, with a sane max_redirects. Set it to false when the terminal response must be the first one.
• Set scrape_timeout to 2 * scrape_interval and the module timeout to less than the scrape timeout.
• Relabel to strip the path from instance. Group alerts by service, not by URL.
• Validate the body regex against the actual response on every change. A regex that no longer matches is a silent false positive.

Verification

You should now be able to answer:

• What does the http_2xx module actually verify, and what does it explicitly not verify?
• Why is valid_status_codes: [200] a more disciplined choice than the default “any 2xx”?
• When is follow_redirects: true the right shape, and when is it a configuration mistake?
• Why is a probe against /healthz insufficient as a boundary assertion for a user-visible flow?
• What does probe_failed_due_to_regex tell you that probe_success=0 does not?

Quiz