ICMP Probes

A network engineer adds an ICMP probe to the black-box dashboard and the panel turns green for every host. A service engineer walks to the panel six hours later, sees green, and assumes the hosts are reachable. They try to SSH into one of them; the connection times out. The ICMP probe never claimed the SSH path was open; the panel never implied it. ICMP is the narrowest probe the exporter runs, and it answers exactly one question: does the host reply to ping.

What it is

The icmp module of blackbox_exporter issues an ICMP echo request to the target, waits for a reply, and returns success if any reply arrives within the timeout. There is no port, no service, no port number. The probe answers host reachability on the ICMP path only. The metric shape is the same as every other module: probe_success, probe_duration_seconds, and probe_ip_protocol (since 0.24).

The ICMP module is also the only module in the default catalogue that requires an unusual operating-system capability. Raw ICMP sockets in the Linux kernel are gated behind CAP_NET_RAW. A process that lacks the capability fails to open the socket; the exporter cannot probe; every metric stays at zero.

Why a sysadmin cares

Two operational facts make the ICMP module a first line of defence rather than a replacement for application probes:

• Many problems fail ICMP first. A host that has lost its IP route, a network partition at the switch, a misconfigured default gateway. These failures often affect ICMP before they affect higher-layer traffic. When the ICMP probe turns red, the network engineer has the earliest possible signal.
• ICMP is the cheapest probe. The protocol is two packets and an echo; the exporter does not maintain any state afterwards. A hundred-target ICMP scrape is cheaper than a ten-target TCP scrape because the kernel bypasses most of the TCP machinery.

The probe does not replace the application probe. A green ICMP probe and a red HTTP probe means the host is up and the application is broken; it never means the application works. The lesson exists to keep both facts visible.

How it works

The exporter opens a raw ICMP socket — when the OS permits — and sends an ICMP_ECHO request to the target address. The target host receives the packet at its networking stack, replies with ICMP_ECHOREPLY, and the exporter records the round trip.

  exporter host                  target host
       |                              |
       | --- ICMP type=8 (echo) -->   |
       |     id=exporter_pid,         |
       |     seq=counter             |
       |                              |
       |                              v
       |                       kernel ICMP handling
       |                              |
       | <-- ICMP type=0 (reply) ---  |
       |     same id, same seq        |
       |                              |
       v                              v
   probe_success=1              probe_duration_seconds =
                                 RTT

The exporter uses the unprivileged ICMP sockets variant in Go where possible (since Go 1.16 with net.Dial("ip4:icmp", ...)). Without CAP_NET_RAW, the kernel returns EPERM on socket open; the exporter reports probe_success=0 for every target. A common production mistake is to assume the ICMP probe is failing when in fact it is the capability that is missing.

How to configure it

The ICMP module is among the simplest in the catalogue. There is no payload, no TLS, no follow-redirects.

# /etc/blackbox/blackbox.yml
modules:

  # Default ICMP probe, IPv4 first, IPv6 fallback.
  icmp_router:
    prober: icmp
    timeout: 2s
    icmp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

  # IPv6-only probe for an environment where IPv4 is being
  # decommissioned. No fallback; ip_protocol_fallback false.
  icmp_v6_router:
    prober: icmp
    timeout: 2s
    icmp:
      preferred_ip_protocol: ip6
      ip_protocol_fallback: false

The corresponding scrape job:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_icmp_hosts
    metrics_path: /probe
    params:
      module: [icmp_router]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['core-router-1.example.internal']
        labels:
          service: network-core
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: module
        replacement: icmp_router

Granting CAP_NET_RAW

In Docker 28.x, the capability is granted per-container:

# docker compose fragment
services:
  blackbox_exporter:
    image: prom/blackbox-exporter:0.26.0
    cap_add:
      - NET_RAW
    # Or the broader choice (not recommended):
    # privileged: true

In Kubernetes / Podman / systemd-nspawn, the equivalent declaration is capabilities.add: ["NET_RAW"] (Compose) or SecurityContext.capabilities.add: ["NET_RAW"] (Kubernetes).

How to validate it

# 1. Sanity: does the host answer ping from the exporter host?
ping -c 4 -W 2 core-router-1.example.internal
# PING core-router-1.example.internal (10.20.4.1) 56(84) bytes of data.
# 64 bytes from 10.20.4.1: icmp_seq=1 ttl=64 time=0.612 ms
# ...
# 4 packets transmitted, 4 received, 0% packet loss, time 3004ms

# 2. The probe, run by hand.
curl -sf "http://blackbox:9115/probe?module=icmp_router&target=core-router-1.example.internal" \
  | grep -E '^probe_'
# probe_duration_seconds 0.001
# probe_ip_protocol 4
# probe_success 1

# 3. Compare to a misconfigured probe (target unreachable).
curl -sf "http://blackbox:9115/probe?module=icmp_router&target=10.255.255.1" \
  | grep -E '^probe_'
# probe_duration_seconds 2.001   # hit the timeout
# probe_success 0

# 4. The capability check on the host.
getcap /usr/local/bin/blackbox_exporter
# (no output means no capabilities; expect cap_net_raw+ep)

# 5. The capability check inside a container.
docker exec blackbox cat /proc/1/status | grep ^Cap
# CapInh: 0000000000000000
# CapPrm: 0000000000002000     # CAP_NET_RAW is bit 13
# CapEff: 0000000000002000
# CapBnd: 00000000a80425fb
# CapAmb: 0000000000000000

If the exporter cannot open a raw ICMP socket, every ICMP probe returns probe_success=0. If the exporter host can ping the target but the probe returns 0, the capability is missing.

How it can fail

1. Missing CAP_NET_RAW. The most common ICMP failure. The exporter cannot open the raw socket; every probe returns 0 from the moment the process started. Symptom: every probe probe_success=0, ping from the host works fine.

2. ICMP dropped at the perimeter firewall. Many networks drop ICMP at the egress ACL for security reasons. The exporter host gets no reply, the probe times out, the target is fine. Symptom: probes red against targets outside the perimeter; internal probes green.

3. ICMP rate-limited. Some providers rate-limit ICMP responses per second. A scrape interval of 5 seconds against a hundred targets provokes the limiter. Symptom: probes return probe_success=1 for the first few scrapes, then 0 until the limiter resets.

4. Double path obfuscation. Two ICMP probes from different exporter hosts take two different paths to the same target. One is green, the other red. Neither is wrong about its own path. Symptom: alerts fire on one exporter only; the team does not know which is the user path.

5. The probe answers a question the SLO does not ask. A team writes an SLO: “database is healthy.” They bind it to an ICMP probe against the database host. The host is green; the database is rejecting sessions. Symptom: SLO is up; the user is down.

6. Latency drift hidden in the success metric. A network path degrades from 1 ms to 50 ms. The probe keeps returning probe_success=1. The SLO does not alert. Symptom: the panel shows green; a slow brownout is in progress.

7. IPv6-only target with IPv4-only exporter host. The target replies only to ICMPv6. The exporter host has no IPv6 route. Symptom: probe 0; the host internal resolver returns AAAA, the dial fails; the exporter has logged the socket open.

How to troubleshoot it

Order matters; an ICMP failure has a small number of distinct causes and the answer is almost always in the first three steps.

1. Read the exporter stderr. docker logs blackbox 2>&1 | tail -50. Look for socket: permission denied, protocol not available, or network is unreachable.
2. Validate the capability on the host. getcap and /proc/1/status. The CapEff bit for CAP_NET_RAW is bit 13; the value 0x2000 is what you expect.
3. Validate ping from the exporter host. If ping works and the probe does not, the problem is the exporter, not the network. If ping fails, the problem is upstream.
4. Compare ICMP to TCP probes against the same target. Both green means the path is healthy. ICMP green, TCP red means the host answers ping but the port is filtered. ICMP red, TCP green is rare; suspect a firewall rule.
5. Compare two ICMP exporters from different networks. Both green means the target answers. Only one green means one of the paths is impaired.
6. Compare probe_duration_seconds to baseline. A rise from 1 ms to 50 ms is a brownout; treat the probe as a latency monitor, not just a binary liveness monitor.
7. Disable the probe temporarily. When the ICMP probe is firing alerts that are noise (a known blocked path), silence it in the alert rule with a dedicated label, not by removing the probe entirely.

Security implications

Raw ICMP sockets are a primitive an attacker would also value. A process holding CAP_NET_RAW can use the local IP address to forge packets and impersonate other hosts. The principle of least privilege says: give the exporter the narrow capability it needs and not the broader NET_ADMIN. Do not run the container as privileged.

The probe reveals internal host topology. A probe that lists internal IP addresses for everyone with read access to Prometheus is an information disclosure. The exporter’s HTTP endpoint is not authenticated; restrict access at the network boundary.

ICMP itself is a denial-of-service vector. A misconfigured scrape interval at low seconds against a high-cardinality target set is a small but real flood. The default thirty-second interval is reasonable; five seconds is the boundary where operator care is required.

Performance implications

The ICMP probe is the cheapest probe the exporter offers — two packets per scrape, no state, no kernel state beyond the socket. A hundred-target scrape at thirty seconds is roughly two hundred socket opens per minute. The exporter handles this without ceremony.

Two pressure points exist. The first is the OS limit on raw sockets per process. A bug or a leak that fails to close the socket exceeds the limit; the next open fails. The exporter is small enough that this is rarely a problem in practice but it is not zero.

The second is the scrape budget. A scrape interval too low against too many ICMP targets is wasteful — the kernel ICMP stack is the bottleneck, not the exporter. The cost-effective shape is one exporter host in each region, each with a dedicated ICMP probe job, each scrape at thirty seconds or longer.

Production guidance

• Always grant CAP_NET_RAW. Never run as root in production; the capability is the narrowest primitive.
• Run the ICMP probe from a host in a different network zone than the targets. The signal value is the external path, not the local path.
• Run at least two ICMP exporters in different regions. The disagreement between them is the signal you actually want.
• Compare ICMP to TCP probes for the same target. The difference between them names the failure mode.
• Track probe_duration_seconds and graph it. Alert on drift, not just on success.
• Do not bind ICMP success to a service SLO. ICMP success is a network SLO; service health is a different question.

Verification

You should now be able to answer:

• Why does the ICMP module require CAP_NET_RAW and what symptom arises when the capability is missing?
• When is ICMP latency drift a brownout that precedes a failure, and why does a binary probe_success not surface it?
• Why is an ICMP probe unsuitable as the SLO for a database or application?
• Why might two ICMP exporters in different regions disagree while both probes are honest about their own path?

Quiz