Lab: Static, Default, Recursive, and Blackhole Routes

Objective

By the end of this lab you will have installed every static-route form VyOS 1.5 LTS offers — direct, indirect, floating, ECMP, blackhole, reject and staged — on a three-router topology, and proved for each one which route the kernel actually forwards on. Every claim gets two outputs, not one: show ip route for what FRR believes and ip route show for what the kernel does. Those two can disagree, and a reader who checks only the first has checked nothing.

Architecture

Three VyOS routers, three point-to-point segments, one “customer” network on r3 that r1 must learn how to reach. The triangle matters: r1 has two physically distinct ways to reach r3, which is what makes floating routes, ECMP and recursive resolution demonstrable on one topology rather than three.

                              r2
                     +--------------------+
                     | eth1          eth2 |
                     +--------------------+
                      .2                .9
                       |                 |
      LAB-A            |                 |            LAB-C
   198.51.100.0/30     |                 |     198.51.100.8/30
                       |                 |
                      .1                .10
        +-------------------+     +--------------------+
        | eth1              |     |               eth2 |
        |         r1        |     |         r3         |
        | eth2         .5   |     |  .6          eth1  |
        +-------------------+-----+--------------------+
             |         LAB-B  198.51.100.4/30      |
          dum0                                    dum0
      192.0.2.1/32                          203.0.113.1/24

eth0 on each router is a management port on whatever bridge your hypervisor already uses. It carries your console fallback and nothing else; no task in this lab configures it, and no route in this lab points at it.

The addresses are all from the ranges RFC 6890 reserves for documentation, so nothing here can collide with a real network if a link is accidentally bridged somewhere it should not be.

Requirements

• A hypervisor with roughly 6 GiB of free RAM and 30 GiB of free disk. Each router needs 1 GiB of RAM and 8 GiB of disk as a floor; the qm and virt-install templates in vyos-iv-02-vm-deployment allocate more.
• The VyOS 1.5 LTS ISO. Every command is written against the 1.5 tree.
• Three isolated layer-2 segments — bridges with no physical port.
• Console access to all three routers, per the callout above.
• No Internet access. Nothing here resolves DNS or reaches a public address.

Scenario

You have inherited a small routed estate. r3 is the site router for a customer network, 203.0.113.0/24. r1 is the edge box you are responsible for, and it has two paths to r3 — one direct over LAB-B, one through r2 over LAB-A and LAB-C. There is no dynamic routing anywhere; every route in the estate was typed by hand, and the person who typed them has left.

Your job is to build r1’s forwarding table from nothing, one route form at a time, and to be able to say — with output — what each addition did.

Tasks

Configuration blocks are written for the [edit] prompt. Where one opens with configure and you are already at [edit], skip that line; blocks opening with set or delete continue the session you have open. bash blocks are operational mode or the hypervisor host, as their comments say.

Task 1 — Build the baseline and capture the starting state

Install the three routers from the ISO and bring them to a bare configuration with a hostname and a management address on eth0. Everything below assumes you are at the [edit] prompt after typing configure.

Before you type a single set, capture what you are starting from. Do this on all three routers.

# Run from operational mode on each router.
JOURNAL="$HOME/lab04"
mkdir -p "$JOURNAL"
show configuration commands > "$JOURNAL/pre-lab-config.txt"
ip route show > "$JOURNAL/pre-lab-kernel-routes.txt"
ip -brief address show > "$JOURNAL/pre-lab-addresses.txt"
cat "$JOURNAL/pre-lab-kernel-routes.txt"

Cleanup compares against those three files. A lab that cannot prove it put the box back has not finished.

r1 gets its two point-to-point addresses and an identity loopback, and no routes at all:

configure
set system host-name r1
set interfaces ethernet eth1 address 198.51.100.1/30
set interfaces ethernet eth1 description 'LAB-A to r2'
set interfaces ethernet eth2 address 198.51.100.5/30
set interfaces ethernet eth2 description 'LAB-B to r3'
set interfaces dummy dum0 address 192.0.2.1/32
set interfaces dummy dum0 description 'r1 identity'
commit
save

r2 is a transit box. It gets both its links and the two routes that let it carry traffic between r1 and r3:

configure
set system host-name r2
set interfaces ethernet eth1 address 198.51.100.2/30
set interfaces ethernet eth1 description 'LAB-A to r1'
set interfaces ethernet eth2 address 198.51.100.9/30
set interfaces ethernet eth2 description 'LAB-C to r3'
set protocols static route 203.0.113.0/24 next-hop 198.51.100.10
set protocols static route 203.0.113.0/24 description 'CR-LAB04 customer LAN via r3'
set protocols static route 192.0.2.0/24 next-hop 198.51.100.1
set protocols static route 192.0.2.0/24 description 'CR-LAB04 r1 identity space'
commit
save

r3 owns the customer LAN and needs a return path to everything r1 might source from:

configure
set system host-name r3
set interfaces ethernet eth1 address 198.51.100.6/30
set interfaces ethernet eth1 description 'LAB-B to r1'
set interfaces ethernet eth2 address 198.51.100.10/30
set interfaces ethernet eth2 description 'LAB-C to r2'
set interfaces dummy dum0 address 203.0.113.1/24
set interfaces dummy dum0 description 'customer LAN'
set protocols static route 192.0.2.0/24 next-hop 198.51.100.9
set protocols static route 192.0.2.0/24 description 'CR-LAB04 return to r1 via r2'
set protocols static route 198.51.100.0/24 next-hop 198.51.100.9
commit
save

Read r3’s last three lines. Both send return traffic through r2 rather than back over LAB-B, deliberately: Task 5 withdraws LAB-B from r1’s side, and a return path that depended on it would die at the same moment.

That makes the normal state asymmetric — traffic r1 sources from dum0 goes out over LAB-B and comes back through r2. ICMP does not mind, but a stateful firewall would see only one direction of the flow, which is the commonest cause of “the ping works and the TCP session does not”. Note also that the static 198.51.100.0/24 does not override r3’s connected /30s: longest prefix wins before distance is consulted at all.

Now prove the starting state on r1. The customer prefix must be unreachable:

$ show ip route

C>* 192.0.2.1/32 is directly connected, dum0
C>* 198.51.100.0/30 is directly connected, eth1
C>* 198.51.100.4/30 is directly connected, eth2

ping -c 3 203.0.113.1

Expect this to fail, and record how it fails, word for word. There are three distinct failures in this lab and they are easy to conflate: a locally originated packet with no matching route at all, which the local socket refuses before anything leaves the box; a packet that matches a route whose next-hop cannot be reached; and a packet that is accepted, forwarded, and discarded silently somewhere on the path. Task 4 produces the second and Task 7 produces the third. Write this one down now so you have all three side by side at the end.

Task 2 — The direct static route, and the two views of it

The direct form names both a next-hop and an egress interface. On a point-to-point link where the neighbour is implied by the wire, it is the form that removes the most ambiguity.

configure
set protocols static route 203.0.113.0/24 next-hop 198.51.100.6 interface eth2
set protocols static route 203.0.113.0/24 description 'CR-LAB04 customer LAN, primary via r3'
commit
save

Now the point of the lab. Ask FRR:

$ show ip route 203.0.113.0/24

S>* 203.0.113.0/24 [1/0] via 198.51.100.6, eth2, weight 1, 00:00:07

Then ask the kernel the same question:

ip route show 203.0.113.0/24
ip route get 203.0.113.10

ip route show prints the FIB entry; ip route get prints the forwarding decision for one address, including the source the kernel would choose. That second one is the habit worth building — it answers “what will happen to this packet” in a way no table dump does.

Both views must agree. A route in show ip route and not in ip route show is in FRR’s RIB and was never installed; nothing forwards on it, and the box will print it at you all day. Read /var/log/frr/zebra.log when that happens.

Prove reachability, then record the path:

ping -c 3 203.0.113.1
traceroute -n 203.0.113.1

The traceroute should show exactly one hop before the destination — you are going straight over LAB-B. Keep that output; Task 3 changes it.

Task 3 — Longest-prefix match beats everything you configured

Add a more-specific route for the lower half of the customer LAN, pointed the other way round the triangle:

configure
set protocols static route 203.0.113.0/25 next-hop 198.51.100.2
set protocols static route 203.0.113.0/25 description 'CR-LAB04 lower half via r2'
commit

Both routes now have distance 1. Predict, in the journal, which route each of these addresses uses, then check:

ip route get 203.0.113.10
ip route get 203.0.113.180
traceroute -n 203.0.113.1

ip route get is the right tool for the first two, because 203.0.113.10 and 203.0.113.180 are addresses inside r3’s LAN with no host behind them — pinging them proves nothing and traceroute to them stops as soon as r3 cannot resolve the last hop. The kernel’s forwarding decision is the fact you are after, and ip route get prints it without sending a packet: 203.0.113.10 resolves out eth1 toward r2, 203.0.113.180 out eth2 toward r3.

The traceroute is to 203.0.113.1, which is a real address on r3, and it now takes two hops instead of the one it took in Task 2 — 203.0.113.1 is inside the /25. Compare it against the capture you kept.

Neither route is “preferred” over the other in any sense a distance value could express. The /25 simply matches more bits, and longest-prefix match runs before administrative distance is consulted at all. This is the single most common reason an operator’s “backup” route carries production traffic: they made it more specific instead of making it more distant.

Remove it again before continuing, so the rest of the lab has one path:

delete protocols static route 203.0.113.0/25
commit
save

Task 4 — Recursive resolution, and the failure that makes it visible

An indirect static route names a next-hop and no interface. The kernel has to find the egress itself by looking the next-hop up in the table — a second lookup, on a route you may not have thought about.

Point r1 at r3’s far side: 198.51.100.10 is r3’s LAB-C address, which r1 has no connected route for.

configure
delete protocols static route 203.0.113.0/24 next-hop 198.51.100.6
set protocols static route 203.0.113.0/24 next-hop 198.51.100.10
commit

Look at it before you fix it. This is the state you need to recognise in an incident:

$ show ip route 203.0.113.0/24

S   203.0.113.0/24 [1/0] via 198.51.100.10 inactive

ip route show 203.0.113.0/24
ping -c 2 203.0.113.1

ip route show prints nothing at all for the prefix. That absence is the evidence — not the word inactive, which is FRR’s opinion, but the empty kernel table, which is the fact.

Now give the next-hop somewhere to resolve:

set protocols static route 198.51.100.8/30 next-hop 198.51.100.2
set protocols static route 198.51.100.8/30 description 'CR-LAB04 reach LAB-C via r2'
commit
save

Re-run the same two commands. The customer route becomes S>*, and its egress is now eth1 — the interface it inherited from the route to its next-hop, not one you ever typed against it. Confirm with traceroute -n 203.0.113.1 that traffic goes through r2.

Return r1 to the direct form before Task 5, and keep the recursive plumbing — Task 6 uses it:

configure
delete protocols static route 203.0.113.0/24 next-hop 198.51.100.10
set protocols static route 203.0.113.0/24 next-hop 198.51.100.6 interface eth2
commit
save

Task 5 — The floating default, and what it is blind to

Give r1 a default route out of each of its two links, with the direct path preferred and the path through r2 held back at distance 210:

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.6 distance 1
set protocols static route 0.0.0.0/0 next-hop 198.51.100.2 distance 210
commit
save

$ show ip route 0.0.0.0/0

S>* 0.0.0.0/0 [1/0] via 198.51.100.6, eth2, weight 1, 00:00:11
S   0.0.0.0/0 [210/0] via 198.51.100.2 inactive

Now withdraw the primary the way a real link failure would: remove the address that gives r1 a connected route toward 198.51.100.6. This is the step that needs the console you confirmed in Requirements.

configure
delete interfaces ethernet eth2 address 198.51.100.5/30
commit

Watch the takeover:

ip route show 0.0.0.0/0
traceroute -n 203.0.113.1

The floating default is installed and the traceroute goes through r2. Record the commit timestamp and that of the first successful traceroute; the gap is your failover time, bounded by how fast the connected route disappears rather than by any protocol.

Note what else happened. The 203.0.113.0/24 route points at 198.51.100.6 out of eth2, and that next-hop is now unreachable, so the customer prefix went inactive at the same moment. Only the default route saved the traffic, and only because r2 knows a path to 203.0.113.0/24. Check show ip route 203.0.113.0/24 and see it.

Restore:

set interfaces ethernet eth2 address 198.51.100.5/30
commit
save

Task 6 — ECMP, and why it is not failover

Flatten the two defaults to the same distance:

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.2 distance 1
commit

ip route show 0.0.0.0/0

Both next-hops are now installed under the one prefix and the kernel load-shares between them. It picks a path per flow, by hashing the 5-tuple — source and destination address, protocol, source and destination port — so a single long-lived TCP session stays pinned to one link for its whole life and gets exactly one link’s worth of bandwidth. Two uplinks is twice the aggregate capacity across many flows, and never twice the speed of one transfer.

To watch the selection you need destinations that only the default route covers. 203.0.113.x is no good — it has its own /24 — so use addresses in 198.51.100.0/24 that fall outside every configured /30, and ask the kernel which next-hop it would pick without sending anything:

for HOST in 198.51.100.200 198.51.100.201 198.51.100.202 198.51.100.203; do
  printf '%s: ' "$HOST"
  ip route get "$HOST" | head -1
done

Some resolve to 198.51.100.2, some to 198.51.100.6, and the split is very unlikely to be even at this sample size. That is the honest shape of ECMP and the reason a “two 1 Gbps uplinks equals 2 Gbps” capacity plan gets escalated to you at some point.

vyos-xii-02-default-route describes the ECMP hash as covering the 5-tuple. What your kernel actually feeds into the hash for forwarded IPv4 traffic is a tunable, and the loop above tells you what moves the selection on the box in front of you: vary the destination and watch it move, then vary only a source port and see whether it does. Quote what you measured, not the rule.

Put the backup back where it belongs:

set protocols static route 0.0.0.0/0 next-hop 198.51.100.2 distance 210
commit
save

Task 7 — Blackhole and reject, from the far side

A blackhole route drops matching packets and says nothing. A reject route drops them and answers with ICMP unreachable. The difference is invisible from the router holding the route and extremely visible from the sender, so test it from the sender.

Point a slice of the customer LAN at r2, and have r2 discard it:

On r1:

configure
set protocols static route 203.0.113.192/26 next-hop 198.51.100.2
set protocols static route 203.0.113.192/26 description 'CR-LAB04 blackhole test slice'
commit

On r2:

configure
set protocols static route 203.0.113.192/26 blackhole
set protocols static route 203.0.113.192/26 description 'CR-LAB04 discard, ticket SEC-77'
commit

Confirm on r2 that the kernel really holds a discard entry, and start a capture on r1 before you send anything:

### On r2
ip route show type blackhole

### On r1, in a second session
tcpdump -ni eth1 icmp

### On r1
ping -c 4 203.0.113.200

The ping reports 100% loss and the capture shows four echo requests leaving and nothing coming back. No ICMP, no clue at the sending end whether the destination is down, the path is broken, or somebody made a decision. That silence is the point: for DDoS discard and martian filtering you do not want to spend a return packet on every dropped one.

Now change one word on r2:

On r2:

delete protocols static route 203.0.113.192/26 blackhole
set protocols static route 203.0.113.192/26 reject
commit

Repeat the ping on r1 with the capture still running. Now each echo request draws an ICMP unreachable back, sourced from 198.51.100.2 — r2 names itself as the router that refused the packet. The sender can tell the difference between “no answer” and “refused”, and a traceroute stops cleanly at r2 instead of running to its hop limit.

Remove both test routes:

On r2:

delete protocols static route 203.0.113.192/26
commit
save

On r1:

configure
delete protocols static route 203.0.113.192/26
commit
save

Task 8 — Stage a route with disable

The last form is the one that does nothing on purpose. Write the route for a change that has not happened yet, and mark it disabled:

configure
set protocols static route 10.0.0.0/8 next-hop 198.51.100.2
set protocols static route 10.0.0.0/8 description 'CR-LAB04 staged, activate in window'
set protocols static route 10.0.0.0/8 disable
commit
save

Two commands, two different answers:

show configuration commands | match '10.0.0.0/8'
show ip route static

The configuration holds the route; FRR has never seen it, so it is in neither the RIB nor the FIB. On the change day, delete protocols static route 10.0.0.0/8 disable and commit — the syntax was reviewed days earlier and the only thing happening in the window is the activation.

The mirror-image failure is common enough to rehearse: a route staged and never activated looks healthy in show configuration and blackholes its prefix. When a prefix is missing from the FIB and present in the configuration, show configuration commands | match disable is a five-second check that belongs near the top of your list.

Validation

Work through each of these and confirm the stated result before you call the lab done.

• show ip route 203.0.113.0/24 on r1 reports S>* with next-hop 198.51.100.6 on eth2, and ip route show 203.0.113.0/24 prints the matching kernel entry. Both, not one.
• ip route get 203.0.113.180 names eth2 as the egress. With the Task 3 /25 reinstated it would name eth1 for 203.0.113.10 and still eth2 for 203.0.113.180; you can state why without re-running it.
• show ip route 0.0.0.0/0 shows exactly one starred default and one inactive entry at distance 210.
• Deleting 198.51.100.8/30 while the customer route is in its recursive form empties ip route show 203.0.113.0/24, and restoring it repopulates the entry — you have the before and after in the journal.
• The blackhole capture contains echo requests and no ICMP replies; the reject capture contains an ICMP unreachable sourced from 198.51.100.2. Two files, visibly different.
• show configuration commands | match '10.0.0.0/8' returns the staged route and show ip route static does not mention it.
• Your answer sheet names, for 203.0.113.10 and 203.0.113.180 under the Task 3 /25, 203.0.113.200 under the Task 7 blackhole, 198.51.100.10, 192.0.2.1 and 10.1.2.3, which route wins and whether longest-prefix match, administrative distance or next-hop resolution decided it.

Expected Outcome

r1 ends the lab with a static configuration that reads like this, and a forwarding table that agrees with it:

protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 198.51.100.2 {
                distance 210
            }
            next-hop 198.51.100.6 {
                distance 1
            }
        }
        route 10.0.0.0/8 {
            description "CR-LAB04 staged, activate in window"
            disable
            next-hop 198.51.100.2 {
            }
        }
        route 198.51.100.8/30 {
            description "CR-LAB04 reach LAB-C via r2"
            next-hop 198.51.100.2 {
            }
        }
        route 203.0.113.0/24 {
            description "CR-LAB04 customer LAN, primary via r3"
            next-hop 198.51.100.6 {
                interface eth2
            }
        }
    }
}

r2 and r3 are back to their Task 1 baselines. ping -c 3 203.0.113.1 from r1 succeeds over the direct link, and you can state what would happen to it under each of the six changes you made and undid.

Troubleshooting

The route is in show ip route and absent from ip route show. FRR holds it, the kernel does not, nothing forwards on it. The usual cause is an unresolved next-hop: run show ip route for the next-hop address and see whether anything matches. If it resolves and the route is still not installed, read /var/log/frr/zebra.log.

commit succeeds and the route never appears anywhere. Check for disable on it, then check you are on the right box.

Ping fails in one direction only. Forward path, no return path. r3’s 198.51.100.0/24 and 192.0.2.0/24 routes are what make the return work. tcpdump -ni eth1 icmp on r3 says immediately whether the request arrived; if it did, the problem is entirely on the return side.

The floating default never activates. Its distance is equal to the primary’s rather than higher. Two defaults at distance 1 are an ECMP pair, and a pair looks like working failover right up to the moment you need it.

A packet you expected to be blackholed gets a reply. A more-specific route matched first. ip route get on the exact address settles it.

Cleanup

Cleanup returns all three routers to the state you captured in Task 1 — not merely “removes the routes”, which would leave r1 without the interface address Task 5 deleted if you skipped the restore.

Step 1. Remove everything the lab added, on each router in turn:

On r1:

configure
delete protocols static
delete interfaces dummy dum0
delete interfaces ethernet eth1 address 198.51.100.1/30
delete interfaces ethernet eth2 address 198.51.100.5/30
commit
save

On r2:

configure
delete protocols static
delete interfaces ethernet eth1 address 198.51.100.2/30
delete interfaces ethernet eth2 address 198.51.100.9/30
commit
save

On r3:

configure
delete protocols static
delete interfaces dummy dum0
delete interfaces ethernet eth1 address 198.51.100.6/30
delete interfaces ethernet eth2 address 198.51.100.10/30
commit
save

Step 2. Prove the box is where it started, on each router:

JOURNAL="$HOME/lab04"
show configuration commands > "$JOURNAL/post-lab-config.txt"
ip route show > "$JOURNAL/post-lab-kernel-routes.txt"
diff -u "$JOURNAL/pre-lab-config.txt" "$JOURNAL/post-lab-config.txt" \
  && echo 'CONFIG RESTORED'
diff -u "$JOURNAL/pre-lab-kernel-routes.txt" "$JOURNAL/post-lab-kernel-routes.txt" \
  && echo 'FIB RESTORED'

Any difference is something the lab created and did not remove. Read it before you move on; a leftover static route in a shared lab environment is how the next lab’s “impossible” symptom gets born.

Step 3. If the routers were built for this lab and nothing else needs them, destroy the VMs. Copy the configurations off first — they are three small text files and they are your evidence.

### On the Proxmox host, once per router
VMID=101
qm stop "$VMID"
qm destroy "$VMID"

Production notes

Every form in this lab maps to a change you will be asked to make, and each carries a different amount of risk in a window.

Adding a static route is close to free until the prefix overlaps something already installed, at which point longest-prefix match silently re-points traffic. The pre-change check is ip route get against a representative address inside the new prefix, run before the commit.

Deleting a static route is where outages come from, and Task 4 is why: the route you are deleting may be resolving a next-hop for a route that never mentions it. Grep the configuration for next-hop addresses inside the prefix first.

Changing distance reads like metadata and is a traffic move. Taking a distance from 210 to 1 shifts every packet for that prefix onto another link at once. Use commit-confirm — see vyos-lab-03-commit-rollback — for anything touching the path you are connected over.

Holding is a first-class option. A degraded-but-working primary with a tested floating backup is often best left alone until the window. Give the hold an owner and an end time in the ticket; a hold without an expiry is how a temporary route survives two years.

What You Learned

• Two views, and they can disagree. show ip route is FRR’s intent; ip route show is what forwards. You saw a route in the first and not the second, and you know which to trust.
• Longest-prefix match runs before administrative distance. The /25 beat the /24 without either carrying a preferred distance, and no distance value could have changed that.
• Recursive routes have invisible dependencies. A customer prefix went down because you deleted an unrelated-looking /30. That is why every static route carries a description naming what depends on it.
• A floating static is blind past the next hop. It failed over when the connected route vanished, and would not have moved at all if r3 had died with the link still up.
• ECMP is aggregate capacity, not per-flow speed, and not failover.
• Blackhole and reject differ only at the sender. Same drop, one with a diagnosis attached; the choice is about who is on the other end.