Network Probes

The application is up. The node metrics are green. The customer in Frankfurt cannot reach it. The platform says everything is fine; the user says nothing works. The metric that resolves the disagreement is the one running on a host that is not in the data centre.

This is what synthetic probes are for. They are the platform’s external eyes.

What it is

Network probes, in this lesson, are synthetic checks issued by blackbox_exporter from a controlled set of source locations against a target set. The two probe families this lesson covers:

• ICMP probes. A single ICMP echo request to a host. The blackbox_exporter icmp module requires CAP_NET_RAW or root on the exporter host. The probe answers “is the host reachable at the IP layer?”
• TCP probes. A single TCP connect() to a host:port. The tcp_connect module answers “is the port accepting connections?”

Both probes return probe_success (1 on success, 0 on failure) and probe_duration_seconds (total wall-clock duration of the probe including DNS resolution, connect, and TLS if applicable). HTTP and DNS probes are covered in their own lessons; the probe mechanics and topology discipline are the same.

The canonical alternative is a managed synthetic service (Grafana Synthetic Monitoring, Pingdom, Catchpoint). The exporter is the right approach when the team already runs Prometheus and the probe sources can run as a small fleet of containers. The managed service is the right approach when the team needs probe sources in many geographies or from inside managed browsers.

Why a sysadmin cares

Three production failure classes disappear the day the probes are in place:

• Route blackouts. A peering session drops, a BGP route withdraws, a CDN edge goes offline. The application is up; the user-facing URL is not. Internal white-box metrics cannot see this. The probe that catches this is an HTTP probe from outside the application’s network (covered in the HTTP lesson).
• External dependency failure. A managed Postgres, an S3-compatible object store, a third-party API. The application is up; the dependency is not reachable from the production path. The probe that catches this is a TCP connect to the dependency’s port from a host that uses the same egress path.
• Silent configuration drift. A 503 that only manifests for one percent of users because a default backend fell out of a load-balancer pool at 02:00. The probe catches it within seconds because the probe path is the same path the user takes.

The probes are a load-bearing component of the observability platform. They are also small, run in single containers, and cost almost nothing in CPU. The reason to skip them is “we have internal monitoring.” The cost of skipping them is the next incident where the user is right and the internal panel is wrong.

How it works

The exporter exposes one HTTP handler, /probe. Prometheus sends a request with two query parameters: target and module. The exporter executes the probe, collects per- protocol metrics, and returns them in Prometheus exposition format.

   prometheus.yml
        |
        v
   scrape job: blackbox_icmp_external
        |
        |  __param_target rewritten from __address__
        |  __param_module = icmp_router
        |
        v
   blackbox_exporter :9115/probe?target=203.0.113.10&module=icmp_router
        |  ----- module: icmp_router ----
        |
        |  ICMP echo request
        |  timeout 2s
        |
        v
   probe_success 1
   probe_duration_seconds 0.014

The scrape job drives the cadence. The exporter drives the protocol details. Prometheus stores the result alongside every other metric in the platform.

The ICMP module

The ICMP module opens a raw ICMP socket on the exporter host, sends an echo request, and waits for the reply. The module requires CAP_NET_RAW capability on the exporter container or root on the host. Without the capability, the module fails to initialise and the probe returns probe_success=0.

The module exposes the standard metrics:

• probe_success — 1 if any reply was received within the timeout.
• probe_duration_seconds — wall-clock duration of the round-trip.
• probe_icmp_duration_seconds — the time spent waiting for the reply after sending the echo request.

The TCP connect module

The TCP module opens a TCP connection to target:port. No application handshake. Success is a successful connect (the three-way handshake completed). The module exposes:

• probe_success — 1 if the connect succeeded.
• probe_duration_seconds — wall-clock duration including DNS resolution, connect, and any TLS if the target is HTTPS.

How to configure it

Three layers are required to put a probe into production: the exporter’s own config, the scrape job on Prometheus, and the relabeling layer that maps each probe target to a high- cardinality, low-cardinality label set.

1. The exporter config: blackbox.yml

# /etc/blackbox/blackbox.yml
modules:
  # ICMP probe for host liveness. Requires CAP_NET_RAW.
  icmp_router:
    prober: icmp
    timeout: 2s
    icmp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: false

  # ICMP probe for an external dependency.
  icmp_dependency:
    prober: icmp
    timeout: 3s
    icmp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

  # TCP connect for database-port reachability.
  tcp_connect_pg:
    prober: tcp
    timeout: 3s

  # TCP connect for an HTTPS frontend (TLS is automatic).
  tcp_connect_edge:
    prober: tcp
    timeout: 5s

  # TCP connect for an internal service.
  tcp_connect_internal:
    prober: tcp
    timeout: 2s

Each module is independent. A module can be reused by many scrape jobs. Name the module after the question, not the protocol: tcp_connect_pg is clearer than tcp1.

2. The scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_icmp_external
    metrics_path: /probe
    params:
      module: [icmp_router]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - 203.0.113.10
          - 203.0.113.11
          - 198.51.100.5
        labels:
      service: external_dependency
          env: prod
          region: eu-west-1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [service]
        target_label: service
      - target_label: module
        replacement: icmp_router
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        replacement: '${1}'
        target_label: instance

  - job_name: blackbox_tcp_connect
    metrics_path: /probe
    params:
      module: [tcp_connect_pg]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - pg-primary.internal:5432
          - pg-replica.internal:5432
        labels:
          service: postgres
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [service]
        target_label: service
      - target_label: module
        replacement: tcp_connect_pg
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        replacement: '${1}'
        target_label: instance

The pattern is the same as every other blackbox probe. The __param_target rule rewrites the address; the module name is stamped via replacement; the instance label is normalised.

3. Validation of the exporter config

# The exporter has no built-in check; the binary refuses to
# start with an invalid config, which is itself a check.
blackbox_exporter --config.check --config.file=/etc/blackbox/blackbox.yml
journalctl -u blackbox_exporter --since '5 min ago' | tail -20

# Validate the Prometheus side.
promtool check config /etc/prometheus/prometheus.yml

How to validate it

Three layers must be confirmed: the exporter is up, the scrape is flowing, and the probe actually exercises the path you think it does.

# 1. The exporter is up. Reads /metrics from the exporter,
#    not from the probe.
curl -sf http://localhost:9115/metrics | grep -E '^blackbox_'
# blackbox_exporter_build_info{version="0.26.0"} 1
# blackbox_exporter_config_last_reload_successful 1
# blackbox_exporter_probe_total 18

# 2. Smoke-test the probe manually before trusting Prometheus.
curl -sf "http://localhost:9115/probe?module=icmp_router&target=203.0.113.10" \
  | grep -E '^(probe_success|probe_duration_seconds)'
# probe_duration_seconds 0.014
# probe_success 1

# 3. Confirm Prometheus is receiving the series.
up{job="blackbox_icmp_external"}
# {instance="203.0.113.10", job="blackbox_icmp_external", service="external_dependency"} 1

probe_success{service="external_dependency"}
# 1

# 4. Confirm the exporter has CAP_NET_RAW for the ICMP module.
getcap /usr/bin/blackbox_exporter
# /usr/bin/blackbox_exporter cap_net_raw=ep
# Or, in a container:
docker inspect blackbox-exporter | jq '.[0].HostConfig.CapAdd'
# ["CAP_NET_RAW"]

# 5. The /-/ready endpoint of the exporter reports modules loaded.
curl -sf http://localhost:9115/-/ready
# ready

If probe_success is 1 and probe_duration_seconds is non-zero, the probe is exercising the right path. If probe_success is 0, inspect the exporter stderr log and the relay metric for the underlying reason.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. ICMP blocked upstream. Many cloud providers and most corporate firewalls drop ICMP beyond the perimeter. The probe gets probe_success=0 for healthy hosts. Symptom: probe_success{module="icmp_router"}=0 while every application-level probe is green.

2. Single exporter host outage. Every probe is red at once. Symptom: up{job="blackbox_*"}=0 for the exporter itself; every target across every module is red.

3. Module timeout shorter than the slowest legitimate response. A 2-second ICMP timeout on a 300 ms satellite link. The probe fires for: 1m before the reply arrives. Symptom: probe_success=0 with probe_duration_seconds close to the timeout value.

4. CAP_NET_RAW missing. The ICMP module fails to initialise. Symptom: every ICMP probe returns probe_success=0; the exporter log shows “operation not permitted” on socket open.

5. TCP connect to a host that drops SYNs. A firewall drops SYNs silently (no RST). The probe times out. Symptom: probe_success=0 with probe_duration_seconds close to the timeout value.

6. Probe target label cardinality. A scrape job against a thousand IPs with no labels produces a thousand series. The TSDB head block churns. Symptom: Prometheus memory rises; prometheus_tsdb_head_series grows.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the exporter alive? curl http://exporter:9115/-/ready and curl http://exporter:9115/metrics first. A missing blackbox_exporter_build_info series means the exporter is not running.
2. Does Prometheus see the exporter? up{job="blackbox_*"} in PromQL. If up=0, the problem is at the scrape boundary, not at the target.
3. Does the probe itself work? Re-run the exact URL the Prometheus scrape would produce, against the exporter manually: curl "http://exporter:9115/probe?module=...&target=...". The headers and body tell you which boundary failed.
4. Check probe_duration_seconds. A drift from 14 ms to 4 s while probe_success=1 is a brownout that has not yet failed. It precedes the failure.
5. Check the exporter host’s own egress. A curl or ping from the exporter host to the target is the fastest test.
6. Compare modules against each other. If tcp_connect_pg is green and icmp_dependency is red on the same target, the failure is in ICMP filtering, not in reachability.

Security implications

The exporter accepts arbitrary target query parameters when exposed without auth. That single endpoint, unauthenticated, can be coerced into probing any IP or port the exporter host can reach. In a multi-tenant environment that becomes a port-scan primitive. Bind the exporter’s /probe to a network Prometheus can reach but untrusted users cannot, or run it on a private network.

The ICMP module requires CAP_NET_RAW on the host. The capability is the same one that allows the host to send arbitrary ICMP packets, including redirects and address-mask requests that older kernels honour. A multi-tenant host that grants CAP_NET_RAW to the exporter process is granting the ability to spoof ICMP packets from that process’s UID. Use the minimal capability set, and isolate the exporter in its own container or VM.

The metric labels are produced by relabeling. A relabel rule that copies a user-controlled URL fragment into a label is a cardinality attack. Whitelist the labels.

Performance implications

The exporter is cheap per probe. The cost scales with the cardinality of the instance and service label sets, the scrape interval, and the time budget of the slowest legitimate response.

• Probe interval. Default 30 s is reasonable. Faster intervals (5 s, 10 s) improve time-to-alert on outages but raise scrape load proportionally.
• Timeout. Each timeout is a goroutine held in the exporter until the deadline. A misconfigured timeout: 60s on a hundred-target scrape job can occupy the exporter for the entire scrape interval.
• Cardinality. instance=host plus service=... plus module=... is the recommended minimum. Anything more, like full IPs in labels, is a budget problem.
• Timeouts and the scrape budget. scrape_timeout on Prometheus must be longer than any module’s timeout. A module with timeout: 5s scraped every 30 s with scrape_timeout: 5s races against the exporter.

Production guidance

• Run at least two probe sources in different failure domains. A common topology is one exporter per region, per cloud provider, or per egress point.
• Pin the module names that alerts and dashboards reference. Renaming a module without coordinating dashboards and alerts is the same class of break as renaming a node_exporter metric on a hundred hosts at once.
• Use a separate scrape job per target class. Do not lump external dependencies and internal services into one job; the failure modes and the alert routing differ.
• Validate the exporter config with --config.check. Validate the Prometheus side with promtool check config. Both must pass before the change ships.
• Document the ICMP permission model. CAP_NET_RAW is not a capability to grant lightly. Run the exporter with the minimum capability set, ideally in a container with the capability dropped.

Verification

You should now be able to answer:

• What is the operational difference between an ICMP probe and a TCP connect probe?
• Why is a single exporter host a single point of failure for synthetic probes?
• Which capability does the ICMP module require, and how is it set on a containerised exporter?
• How do you decide between a 30-second probe interval and a 10-second interval?
• What is the trade-off between blackbox_exporter and a managed synthetic service?

Quiz