Gateway monitoring and dpinger — the probe, the thresholds, the triggers

A multi-WAN deployment without monitoring is a single-WAN deployment that has not noticed the WAN is dead. Monitoring is what turns the second WAN from a configured-but-idle interface into a real failover path. OPNsense uses dpinger — a lightweight ICMP probe daemon — to measure loss and latency on every configured gateway, and the trigger levels that decide when a gateway is “down” are configurable per gateway. This lesson covers what dpinger measures, how the thresholds interact, and the discipline of picking a monitor IP that actually reflects WAN health.

What dpinger measures

dpinger sends ICMP echo requests (pings) to a configured monitor IP at a regular interval (the probe interval, default 1 second) and tracks three values per gateway:

Packet loss. The percentage of probes that did not return a reply within the timeout. A loss of 0% is healthy; loss above the configured threshold is the trigger for the packet-loss trigger level.

Latency. The round-trip time of the probes that did return. dpinger tracks the average, the running mean, and the high-water mark over a window of samples. Latency above the configured threshold is the trigger for the high-latency trigger level.

Reachability. Whether the monitor IP can be reached at all. If every probe in the window fails, dpinger marks the gateway as offline. The Member Down trigger level fires when the gateway is fully unreachable.

The probe is one ICMP echo per second. At that rate, dpinger adds negligible load to the WAN; the network overhead is dwarfed by normal user traffic. The cost is not bandwidth; it is the choice of monitor IP — the IP has to be reachable through the gateway and not reachable when the gateway is down.

Picking the monitor IP

The monitor IP is the most consequential decision in gateway monitoring. The IP has to:

1. Be reachable through the gateway. The probe must reach the monitor IP via the gateway being monitored. An IP that is reached through a different gateway gives a false reading.
2. Reflect WAN health, not endpoint health. A monitor IP that is down for reasons unrelated to the WAN makes the gateway look down when the WAN is fine.
3. Be stable. The monitor IP cannot move; a monitor IP that disappears when the upstream changes routing produces spurious failovers.

Common choices and their trade-offs:

The four trigger levels

The trigger level decides what makes a gateway “down”. OPNsense offers four:

Member Down. Fires when the gateway has 100% packet loss. The most conservative trigger: the gateway is unusable. The advantage is that false positives are rare. The disadvantage is that a degraded gateway (50% loss, high latency) is treated as healthy until the failure is complete.

Packet Loss. Fires when the loss exceeds the configured threshold (default 5%). Catches degraded gateways. The disadvantage is that brief loss spikes (a single bad minute) can trigger failover unnecessarily.

High Latency. Fires when latency exceeds the configured threshold (default 200 ms). Catches gateways that are technically up but performing poorly. The disadvantage is that the threshold is WAN-specific; a fibre WAN should be 5-20 ms, an LTE WAN may be 50-100 ms, a satellite WAN may be 600+ ms. The default of 200 ms fits some WANs and not others.

Packet Loss or High Latency. Fires when either condition is met. The most aggressive trigger; catches both loss and latency problems.

The right choice depends on the WAN technology and the deployment’s tolerance for failover churn:

$ sockstat -46 -p 53 -c -P dpinger

USER     PROCESS    PID   PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     dpinger    5821  icmp4  192.0.2.1:51234       *:*
root     dpinger    5822  icmp4  192.0.2.1:51235       *:*

The advanced tuning knobs

Five advanced knobs are exposed per gateway under System → Gateways → Configuration → Advanced:

Latency low / high thresholds. The thresholds for the latency trigger. The low threshold is the latency below which the gateway is considered healthy; the high threshold is the latency above which the gateway is considered degraded.

Packet loss low / high thresholds. Same idea for loss.

Probe interval. How often dpinger sends a probe. Default 1 second. Lower intervals catch failures faster but add more probe traffic; higher intervals save bandwidth but detect failures more slowly.

Down. The number of seconds of failed probes before the gateway is marked offline. Default 10 seconds. Lower values trigger failover faster; higher values tolerate brief outages without failover.

Lost Probe Delay. The number of probe samples after which loss is computed. Higher values smooth the loss percentage over more samples; lower values make loss more reactive to recent failures.

The right values depend on the WAN. A fibre WAN with consistent latency can use a 5-second down threshold and a 1-second probe interval; a wireless WAN with variable latency benefits from a 10-second down threshold and a longer lost probe delay to smooth out momentary signal loss.

False failover: the most common gateway monitoring incident

False failover is when the monitoring marks a gateway down when the WAN is fine. Three patterns cause it:

The monitor IP is unreachable for endpoint reasons. The monitor IP (say, 8.8.8.8) is up but the path from the firewall to it goes through a router that briefly stopped forwarding. dpinger sees loss; marks the gateway down; triggers failover. The WAN is fine.

The monitor IP is too far away. The probe goes through the WAN, through the ISP’s network, through a couple of upstream routers, to the monitor IP. A failure anywhere on that path is reported as a WAN failure. The fix is a monitor IP that is as close to the gateway as possible (the ISP’s gateway IP, ideally).

The probe interval is too short. A 100-millisecond interval averages over very few samples; a single bad probe is 100% loss for that window. The fix is a longer interval or a longer down threshold.

The diagnostic for false failover is the dpinger log and the gateway overview page. The overview shows the loss and latency history; the operator can see whether the gateway has a chronic problem (real failure) or a sporadic problem (false failover).

Reading the gateway overview

The System → Gateways → Configuration page shows every gateway and its current state. The columns:

A healthy gateway shows Online with low RTT and 0% loss. A gateway in trouble shows Warning or Offline with elevated RTT or loss. The page is the operator’s first stop when investigating a failover.

Summary

• dpinger probes each gateway with ICMP at a configurable interval and tracks loss, latency, and reachability.
• The monitor IP must be reachable through the gateway and reflect WAN health, not endpoint health.
• The four trigger levels (Member Down, Packet Loss, High Latency, Packet Loss or High Latency) match different WAN technologies.
• Advanced knobs (thresholds, probe interval, down delay) let the operator tune the trigger to the WAN’s character.
• False failover is the most common production incident; it is usually caused by a monitor IP that is too far away, by an over-aggressive threshold, or by an over-short probe interval.
• The gateway overview page is the operator’s first stop for monitoring health; review it weekly during the first month of a new deployment.