WAN health check — the three test types VyOS ships, the counters that replace a threshold, and the targets worth probing

The health check is the part of a multi-WAN build that decides whether a circuit is alive. Everything else — the routes, the distances, the translation — either works or produces an obvious error at commit. The health check is the part that reports healthy while the circuit is dead, silently, for months, until the day it matters.

It is also the part where the platform gives you less than people expect, so this lesson starts with what is actually there.

What VyOS 1.5 gives you, and what it does not

Health checking on VyOS lives in exactly one place: set load-balancing wan interface-health. There is no service health-check, and the health check is inseparable from the load balancer — if you are building failover with a floating static route instead, your detector is BFD or interface carrier, and none of this lesson applies to it.

Under interface-health you get:

The three test types

ping

An ICMP echo request to target. The simplest test, the lowest overhead, and the one with the most ways to be fooled.

configure
set load-balancing wan interface-health eth0 nexthop 203.0.113.1

set load-balancing wan interface-health eth0 test 10 type ping
set load-balancing wan interface-health eth0 test 10 target 203.0.113.1
set load-balancing wan interface-health eth0 test 10 resp-time 3

set load-balancing wan interface-health eth0 test 20 type ping
set load-balancing wan interface-health eth0 test 20 target 198.51.100.20
set load-balancing wan interface-health eth0 test 20 resp-time 3

set load-balancing wan interface-health eth0 failure-count 3
set load-balancing wan interface-health eth0 success-count 5

compare
commit

Two tests, deliberately. A probe to the handoff and a probe past it fail for different reasons and neither is a superset of the other: the handoff probe cannot see a provider whose own upstream has failed, and the distant probe cannot tell your circuit failing from that target failing.

How several tests on one interface combine — whether all must pass or any one is enough — is worth establishing on your own image with a deliberate single-target failure before you depend on the answer. That is a five-minute experiment during the build and an argument during an incident.

ttl

Sends a probe with a limited time-to-live and looks for the ICMP time-exceeded reply that the hop at ttl-limit returns when it discards it.

set load-balancing wan interface-health eth1 nexthop 198.51.100.1
set load-balancing wan interface-health eth1 test 10 type ttl
set load-balancing wan interface-health eth1 test 10 target 198.51.100.20
set load-balancing wan interface-health eth1 test 10 ttl-limit 2

This is the answer to a handoff router that has ICMP echo turned off but still forwards traffic, and to a provider who rate-limits echo replies to their own edge while transiting everything else normally. A ping test against such a handoff produces false positives forever; a ttl test asks a different question — is there still a router two hops along this path — and gets a different answer.

It has its own precondition: the intermediate hop must return ICMP time-exceeded. Some do not, and for those the test fails permanently and the circuit is declared dead while it is carrying traffic. Confirm the behaviour with traceroute from the router before you rely on it.

user-defined

A script. VyOS runs it and reads its exit status: zero is a pass, anything else is a failure. This is where every check the platform does not ship lives — HTTP status codes, TCP connect, DNS resolution, a latency ceiling, a check against your own monitoring.

set load-balancing wan interface-health eth0 test 30 type user-defined
set load-balancing wan interface-health eth0 test 30 test-script /config/scripts/wan-http-check.sh

Put the script under /config/scripts/, which survives an image upgrade. A script in /usr/local/bin does not, and a health check that silently stops existing after an upgrade is a worse failure than the one it was written to catch.

The script owns three things VyOS will not do for it: choosing which interface to leave by, bounding its own runtime, and deciding what counts as failure.

#!/bin/bash
# /config/scripts/wan-http-check.sh
# Exit 0 means the circuit is healthy. Any other status is a failed test.
# Substitute your own values before deploying:
WAN_IF=eth0
PROBE_URL="http://198.51.100.20/healthz"
MAX_SECONDS=3

set -u

# --interface pins the request to the circuit under test. Without it the
# request follows the main routing table and can leave by the other WAN,
# which would make this script report on a circuit it never touched.
if curl --interface "$WAN_IF" \
        --max-time "$MAX_SECONDS" \
        --fail --silent --show-error \
        --output /dev/null \
        "$PROBE_URL"; then
  exit 0
fi

exit 1

The counting model

resp-time, failure-count and success-count are the whole of the tuning surface, and they do different jobs.

resp-time bounds one test. It is how long the engine waits for an answer before recording that test as failed. Set it explicitly rather than inheriting a default, because the default belongs to the release and the right value belongs to your circuit: long enough that a normally-loaded link does not fail, short enough that a dead one is noticed.

failure-count is consecutive failures before the circuit is declared unusable. success-count is consecutive successes before it is declared usable again.

Choosing a target

The engine sends each interface’s tests through that interface’s own routing table — that is what nexthop is for — so the classic story of a probe escaping out the other circuit and reporting a dead link as healthy does not apply here the way it does to a ping you type at the CLI. Two other target problems very much do.

A shared target is a shared dependency. Point both interfaces at the same well-known public address and you have handed one third party the ability to declare both of your circuits dead at the same moment. Use different targets, in different networks, for different circuits.

A distant target cannot distinguish two failures. A probe to a server five networks away fails when your circuit fails and also when that server, or anything between, fails. This is not a reason to avoid distant targets — they are the only thing that sees a provider whose own upstream is down — it is a reason to pair them with a next-hop probe, so the two together say which of the two happened.

Prefer an address you have a relationship with. The provider’s handoff, the provider’s resolver, the far end of your own site-to-site link. Something you can ring somebody about when it starts behaving oddly.

Verifying that the check is doing anything

show wan-load-balance
show wan-load-balance connection

Read show wan-load-balance for three things: that the engine knows about every interface you configured, that it currently considers each one usable or not, and that its tests are producing results rather than sitting at zero. The exact column layout has changed between releases, so read it rather than parsing it; if you want an alarm on a state change, attach a script with set load-balancing wan hook and let the engine call you.

No sample output is reproduced here. The one field that matters is whether your interface is usable right now, and no printed example can tell you that — while a printed example is very good at persuading a tired operator that a column they cannot find must be somewhere off-screen.

Then confirm the check is attached to something. Health tests that report perfectly and steer no traffic are the commonest defect in this whole subject:

show configuration commands | match "load-balancing wan rule"
sudo ip rule list

If the first returns nothing, the balancer is an accurate monitoring system attached to no traffic. Part XXXIX-02 covers the rule.

Production failure modes

1. Attached to nothing. interface-health is configured and correct; no rule exists; the tests change nothing.
2. False positive from echo policing. The handoff deprioritises or rate-limits ICMP echo under load. The circuit is fine and the ping test fails. Fix with a ttl test, a different target, or a higher failure-count — in that order of preference.
3. Permanent false negative from a ttl test. The hop at ttl-limit does not return ICMP time-exceeded, so the test can never pass. Confirm with traceroute before deploying it.
4. Shared target outage. Both circuits probe the same public address, that address has a bad afternoon, and both circuits are declared unusable at once.
5. A test script that hangs. No timeout on the network call, no exit status, no result. This looks like a health check that has stopped rather than one that is failing.
6. A script lost to an upgrade. test-script points outside /config/, the image is upgraded, and the check quietly stops existing.

Rollback

delete load-balancing wan interface-health eth0 test 20
delete load-balancing wan interface-health eth0
compare
commit
save

Deleting a single test is the smaller change and usually the right one: it removes the probe that is misbehaving and leaves the rest of the detection in place. Deleting the whole interface-health block leaves the interface in use whatever its state, which is the pre-change behaviour of a router with no health checking — degraded, but predictable, and worth choosing deliberately over a check that flaps.

rollback 1 then commit reverts the revision; rollback alone loads the candidate and changes nothing. commit without save survives until the next reboot.

Production discipline

Cross-course references

• Part XXXIX-01 (XXXIX-VyOS-MultiWAN / concept) covers the three multi-WAN mechanisms and where health checking sits among them.
• Part XXXIX-02 (XXXIX-VyOS-MultiWAN / failover) builds the rule that gives these tests something to act on.
• Part XXXII-05 (XXXII-VyOS-BFD / BFD with static) covers the detector for the floating-static design.
• Part LII (LII-VyOS-Troubleshoot) covers the broader troubleshooting methodology.

Quiz