Lab: BGP Traffic Engineering

Objective

By the end of this lab you will have moved traffic in both directions across a dual-homed site, and — more importantly — you will be able to say for each change whose decision you altered. Outbound traffic is yours to decide, and LOCAL_PREF decides it outright. Inbound traffic is decided by other people’s routers, so the best you can do is send them an argument. AS-path prepending is a weak argument that every AS understands; a provider community is a strong argument that only your provider honours. Both are in this lab, and so is the moment where using both at once makes them cancel out.

Along the way you will watch what FRR does with the paths that lose, which is the part of traffic engineering most people never look at.

Architecture

The Lab 11 topology, unchanged: a dual-homed site and two upstreams that also peer with each other.

                      AS 65001  site1
                      router-id 192.0.2.101
                      originates 198.51.100.0/24
                        /                \
      eth1 192.0.2.1/30                   eth2 192.0.2.5/30
            /                                      \
  eth1 192.0.2.2/30                          eth1 192.0.2.6/30
   AS 65010  isp-a                            AS 65020  isp-b
   originates 203.0.113.0/25                  originates 203.0.113.128/25
        \                                          /
      eth2 192.0.2.9/30 ------------------- eth2 192.0.2.10/30

The triangle is what makes traffic engineering demonstrable at all. Each of the three destinations in this topology is reachable two ways, so every router has a genuine choice to make, and every change you make is a change to somebody’s choice.

Requirements

• The completed Lab 11 topology, at the state Lab 11’s Cleanup told you to stop at: three routers, export and import policy on site1, communities 65001:1010 and 65001:1020 applied on import, and isp-a still originating 198.51.100.128/25.
• 6 GB RAM and 24 GB disk in total across three VMs.
• VyOS 1.5 LTS, FRR 10.x. The direction keywords are import and export.
• Console access to all three VMs. Task 3 disables an interface on purpose. Management stays on eth0 and is never touched.
• Your community registry from Lab 11. This lab adds a row to it, and the row is the deliverable that makes the rest reproducible.
• Roughly 100 minutes.

If you no longer have the Lab 11 state, its Task 1 rebuilds the topology from a fresh install and the block below rebuilds the policy on top.

Scenario

The site has two transit links. They are not equivalent: isp-b is the larger and cheaper of the two, and the site wants to use it for essentially everything, keeping isp-a as a backup that carries traffic only when isp-b cannot.

That is one sentence of intent, and it hides two entirely separate engineering problems. Traffic the site sends is decided by site1. Traffic the site receives is decided by every other router on the path, none of which the site controls. The first problem has a clean solution. The second has only arguments of varying strength.

Tasks

Task 1: Establish which upstream is used today, and why

Before changing anything, produce the baseline. For each destination, which upstream does site1 currently use, and which step of the best-path algorithm decided it?

$ show bgp ipv4 unicast 203.0.113.0/25

BGP routing table entry for 203.0.113.0/25
Paths: (2 available, best #1, table default)
65010
  192.0.2.1 from 192.0.2.1 (192.0.2.102)
    Origin IGP, localpref 100, valid, external, best (Shorter AS-Path)
    Community: 65001:1010
65020 65010
  192.0.2.5 from 192.0.2.5 (192.0.2.103)
    Origin IGP, localpref 100, valid, external
    Community: 65001:1020

FRR prints the deciding step in parentheses after best. Here it is Shorter AS-Path — both paths have LOCAL_PREF 100, neither is locally originated, and the tie broke at step four of the ladder. The communities you attached in Lab 11 are visible on both paths, and this is the first place they pay off: you can tell which session admitted each path without cross-referencing a next-hop address against an interface list.

Run the same command for 203.0.113.128/25 and for 198.51.100.128/25, and fill in the table in your journal:

The third row is the interesting one. 198.51.100.128/25 is the mis-origination isp-a has been announcing since Lab 11, and site1’s import filter rejects it — so site1 has no path at all, and the correct entry is “none, filtered on import”. A destination that is absent because you decided it should be absent still belongs in the table, because six months from now the only difference between that and an outage is whether somebody wrote it down.

Task 2: Move outbound traffic with LOCAL_PREF

The intent is “prefer isp-b for everything”. LOCAL_PREF is the right tool: it is compared at step two, before AS-path length, so it decides regardless of how the AS paths compare. It is also the correct scope — it propagates through iBGP, so on a real network every router in AS 65001 inherits the decision rather than each one making its own.

Add the preference to the import route-map that already handles the isp-b session:

configure
set policy route-map FROM-ISP-B rule 10 set local-preference 200
commit-confirm 5

Confirm, then trigger a route-refresh so the routes already received are re-evaluated against the changed policy:

clear ip bgp 192.0.2.6 in

Now re-read the same prefix:

$ show bgp ipv4 unicast 203.0.113.0/25

BGP routing table entry for 203.0.113.0/25
Paths: (2 available, best #2, table default)
65010
  192.0.2.1 from 192.0.2.1 (192.0.2.102)
    Origin IGP, localpref 100, valid, external
    Community: 65001:1010
65020 65010
  192.0.2.5 from 192.0.2.5 (192.0.2.103)
    Origin IGP, localpref 200, valid, external, best (Local Pref)
    Community: 65001:1020

best #2, and the reason is now Local Pref. The winning path is a hop longer and traverses an extra AS, and it wins anyway, because LOCAL_PREF is compared first. Understanding why that ordering exists is the whole point: AS-path length is a proxy for cost that BGP invented because it had nothing better; LOCAL_PREF is your actual commercial and capacity knowledge, and it is placed above the proxy deliberately.

Update your table. Every destination reachable via isp-b should now prefer it, including the ones isp-a originates.

Add it now:

configure
set policy route-map FROM-ISP-A rule 10 set local-preference 100
commit
save

Task 3: Watch what FRR installs, and time the failover

site1 holds two paths to 203.0.113.0/25. Exactly one of them is usable for forwarding. Ask each layer in turn.

$ show bgp ipv4 unicast 203.0.113.0/25 | match 'Paths:'

Paths: (2 available, best #2, table default)

$ show ip route 203.0.113.0/25

Routing entry for 203.0.113.0/25
Known via "bgp", distance 20, metric 0, best
Last update 00:01:41 ago
* 192.0.2.5, via eth2, weight 1

$ ip route show table 254 203.0.113.0/25

203.0.113.0/25 nhid 27 via 192.0.2.5 dev eth2 proto bgp metric 20

Two paths, one route, one nexthop. bgpd runs best-path selection and offers only the winner to zebra; the loser stays in the BGP table as a hot spare and never reaches the kernel. Installing more than one requires BGP multipath to be configured explicitly and requires the candidate paths to be equal in every attribute the algorithm compares before the tie-break — the ECMP lesson covers both conditions and the multipath-relax knob that loosens the second.

This is why “the route is in BGP” and “the route is forwarding” are different sentences, and why a traffic engineering change is not finished until you have looked at the kernel.

Now use the spare. Disable the interface toward isp-b and time how long the topology takes to converge.

configure
set interfaces ethernet eth2 disable
commit-confirm 5

Watch the convergence, then confirm the kernel followed:

$ ip route show table 254 203.0.113.0/25

203.0.113.0/25 nhid 22 via 192.0.2.1 dev eth1 proto bgp metric 20

Note what made this fast: the alternative path was already in the BGP table, already valid, and already evaluated. Selection and installation are local work — no messages had to cross the network. That is the argument for keeping a second session up and filtered rather than holding a backup provider in reserve and turning it on during an incident.

Bring the link back and check that the preference reasserts itself:

configure
delete interfaces ethernet eth2 disable
commit
save

Task 4: Move inbound traffic with AS-path prepending

Outbound is settled. Inbound is a different problem: site1 does not decide it. Every router on the internet that wants to reach 198.51.100.0/24 runs its own best-path algorithm, and site1’s only influence is the shape of the announcements it sends.

Check the current state on the router whose decision you are about to change:

$ show bgp ipv4 unicast 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (2 available, best #1, table default)
65001
  192.0.2.5 from 192.0.2.5 (192.0.2.101)
    Origin IGP, localpref 100, valid, external, best (Shorter AS-Path)
65010 65001
  192.0.2.9 from 192.0.2.9 (192.0.2.102)
    Origin IGP, localpref 100, valid, external

Suppose the site wants inbound traffic to arrive via isp-a instead. The classical tool is to make the announcement toward isp-b look worse, by prepending the local AS to the path several times.

site1’s export policy toward isp-b is currently a prefix-list, so convert it to a route-map — one mechanism per direction, as in Lab 11 — and put the prepend in it:

configure
set policy route-map TO-ISP-B rule 10 action permit
set policy route-map TO-ISP-B rule 10 description "announce only our own space, deprioritised toward isp-b"
set policy route-map TO-ISP-B rule 10 match ip address prefix-list ONLY-MINE
set policy route-map TO-ISP-B rule 10 set as-path prepend 65001 65001 65001
delete protocols bgp neighbor 192.0.2.6 address-family ipv4-unicast prefix-list export
set protocols bgp neighbor 192.0.2.6 address-family ipv4-unicast route-map export TO-ISP-B
commit-confirm 5

Confirm, and verify from both ends. First that site1 is sending what you think:

$ show ip bgp neighbors 192.0.2.6 advertised-routes

   Network          Next Hop            Metric LocPrf Weight Path
*> 198.51.100.0/24  0.0.0.0                  0         32768 65001 65001 65001 i

Total number of prefixes 1

Then that isp-b changed its mind — which is the only result that matters:

$ show bgp ipv4 unicast 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (2 available, best #2, table default)
65001 65001 65001 65001
  192.0.2.5 from 192.0.2.5 (192.0.2.101)
    Origin IGP, localpref 100, valid, external
65010 65001
  192.0.2.9 from 192.0.2.9 (192.0.2.102)
    Origin IGP, localpref 100, valid, external, best (Shorter AS-Path)

$ ip route show table 254 198.51.100.0/24

198.51.100.0/24 nhid 31 via 192.0.2.9 dev eth2 proto bgp metric 20

Three things are worth recording about what just happened.

Prepend your own AS and nobody else’s. set as-path prepend 65001 65001 65001 inserts the local AS. Inserting a number you do not hold is a forgery: it claims a route traversed a network it never touched, it is detectable, and it will get your session shut down.

Three is a convention, not a mechanism. Each prepend adds one to the path length, and it only changes anything for a router that reaches step four with a genuine alternative. If isp-b had a LOCAL_PREF policy preferring its customers — which most transit providers do — your prepends would be compared at step four, long after the decision was already made at step two, and nothing would move. Prepending is an argument, and the other end is free to have already decided.

More is not better. Enough prepends and some networks stop accepting the route at all, on the reasonable grounds that a path that long looks like a loop or a leak. That converts “deprioritise this link” into “become unreachable from part of the internet”.

Now remove the prepend, and read the reason carefully, because it is the most transferable thing in this lab:

configure
delete policy route-map TO-ISP-B rule 10 set as-path prepend
commit
save

With the prepend in place, isp-b’s best path to the site runs through isp-a. BGP does not advertise a path back to the peer it learned it from, so isp-b stops offering isp-a any route to 198.51.100.0/24 at all — and Task 5, which needs isp-a to have a second path to fail over to, would have nothing to work with. Two inbound techniques applied at once did not add up; the first removed the raw material of the second.

That is the general shape of inbound traffic engineering. You are not setting a value, you are influencing a chain of independent decisions, and the chain has feedback in it.

Task 5: Move inbound traffic with a provider community

Prepending works with anyone and works weakly. The strong version requires cooperation: many transit providers publish communities that a customer can attach to a route to control the LOCAL_PREF the provider applies to it. The technique dates to RFC 1998 and is still the standard way a multihomed customer expresses primary and backup.

Play both sides. First, isp-a publishes the offer. Tag 65010:80 means “set LOCAL_PREF 80 on this route”, below the provider’s default of 100:

configure
set policy community-list CUST-LOWPREF rule 10 action permit
set policy community-list CUST-LOWPREF rule 10 community 65010:80
set policy community-list CUST-LOWPREF rule 10 description "customer signal: treat as backup"

set policy route-map FROM-CUSTOMERS rule 10 action permit
set policy route-map FROM-CUSTOMERS rule 10 description "honour the published backup community"
set policy route-map FROM-CUSTOMERS rule 10 match community CUST-LOWPREF
set policy route-map FROM-CUSTOMERS rule 10 set local-preference 80
set policy route-map FROM-CUSTOMERS rule 20 action permit
set policy route-map FROM-CUSTOMERS rule 20 description "everything else at the provider default"

set protocols bgp neighbor 192.0.2.1 address-family ipv4-unicast route-map import FROM-CUSTOMERS
commit-confirm 5

Rule 20 is not optional. A route-map denies anything that matches no permit rule, exactly as a prefix-list does. Without rule 20, attaching FROM-CUSTOMERS would reject every route from the customer that did not carry the community — which is to say, all of them.

Now, on site1, take the offer up. Convert the isp-a export to a route-map and attach the community:

configure
set policy route-map TO-ISP-A rule 10 action permit
set policy route-map TO-ISP-A rule 10 description "announce only our own space; signal backup with the isp-a community"
set policy route-map TO-ISP-A rule 10 match ip address prefix-list ONLY-MINE
set policy route-map TO-ISP-A rule 10 set community 65010:80 additive
delete protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast prefix-list export
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast route-map export TO-ISP-A
commit-confirm 5

Confirm, then verify on isp-a:

$ show bgp ipv4 unicast 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (2 available, best #2, table default)
65001
  192.0.2.1 from 192.0.2.1 (192.0.2.101)
    Origin IGP, localpref 80, valid, external
    Community: 65010:80
65020 65001
  192.0.2.10 from 192.0.2.10 (192.0.2.103)
    Origin IGP, localpref 100, valid, external, best (Local Pref)

The direct path from the customer carries LOCAL_PREF 80. The path via isp-b sits at the default 100. isp-a now sends traffic destined for the site the long way round, through its peer, and it made that decision at step two of the ladder — where prepending never gets a hearing.

Confirm the community is really what did it by asking for the routes that carry it:

$ show bgp community 65010:80

   Network          Next Hop            Metric LocPrf Weight Path
*  198.51.100.0/24  192.0.2.1                0             0 65001 i

Displayed 1 routes and 1 total paths

The * without > is the detail to notice: the path is valid, and it is not the best one. That single character is the difference between “the route is there” and “the route is used”.

Task 6: Read the whole picture back

You have made four changes. Rebuild the table from Task 1 and diff it against what you wrote at the start.

$ show ip route bgp

B>* 203.0.113.0/25 [20/0] via 192.0.2.5, eth2, weight 1, 00:07:12
B>* 203.0.113.128/25 [20/0] via 192.0.2.5, eth2, weight 1, 00:07:12

Both destinations leave via eth2 toward isp-b, which is the outbound intent from the Scenario. Inbound, the site announces itself to isp-a with the backup community, so isp-a reaches it through isp-b; and it announces itself to isp-b unmodified, so isp-b reaches it directly. Traffic in both directions now favours isp-b, with isp-a holding a live, filtered, immediately usable backup path in both directions.

Write the last deliverable: for each of the two inbound techniques, one sentence on what it cannot do and who has to cooperate. That sentence is what you will need in a design review, and it is the part of traffic engineering that does not fit in a configuration.

Validation

• Your before/after table exists for all three destinations, and each row names a best-path step, not just a next hop.
• show bgp ipv4 unicast 203.0.113.0/25 on site1 reports best (Local Pref) with the winning path carrying localpref 200 and a longer AS path than the loser.
• show ip route 203.0.113.0/25 and ip route show table 254 203.0.113.0/25 on site1 each show exactly one nexthop, via eth2, while the BGP table still holds two paths.
• Your journal records the failover: the kernel nexthop before eth2 was disabled, after, and after it was re-enabled.
• During Task 4, show bgp ipv4 unicast 198.51.100.0/24 on isp-b showed best moving to the path via 192.0.2.9, and the kernel FIB on isp-b followed. Both captures are in the journal.
• With the prepend removed and the community applied, show bgp ipv4 unicast 198.51.100.0/24 on isp-a reports best (Local Pref) on the path via 192.0.2.10, and the direct path shows localpref 80 and Community: 65010:80.
• show bgp community 65010:80 on isp-a returns the site prefix with * and no >.
• Your community registry has at least three rows, each naming who sets the value, who honours it, and what it means.

Expected Outcome

A dual-homed site whose outbound traffic prefers isp-b by explicit LOCAL_PREF on both sessions, and whose inbound traffic prefers isp-b because isp-a has been asked, in a language it published, to treat the direct path as backup. Both upstream sessions remain up, filtered and tagged. Disabling either link moves all traffic to the other within one convergence cycle, and you have measured that cycle rather than assumed it.

The topology is the same one Lab 11 left behind; only policy changed. Every policy object is named for what it does and carries a description.

Troubleshooting

LOCAL_PREF appears in the configuration but not on the routes. The routes were received before the route-map changed. Run clear ip bgp <peer> in, substituting the peer address, to trigger a route-refresh. Policy applies at the moment a route is evaluated, not retroactively.

The prepend is configured but the AS path is unchanged. Check the attachment direction — a prepend belongs on route-map export. Then check the route-map’s own logic: the prepend must sit on a rule that permits and matches the prefix. A set clause on a rule whose match never fires does nothing, silently.

isp-b still prefers the direct path after prepending. Compare the two AS-path lengths in show bgp ipv4 unicast on isp-b. Three prepends make the direct path length four against two for the alternate, so if it still wins, something earlier in the ladder is deciding — check localpref on both paths first.

The community does not appear on the receiving router. Confirm it is being set: show bgp community 65010:80 on the sender should list the prefix. If it is set locally and absent remotely, check whether your build sends communities to eBGP peers by default, and check for a set community without additive anywhere in the chain.

All routes from a peer disappear after attaching a route-map. The implicit deny. A route-map rejects anything matching no permit rule — add a final bare action permit rule, as FROM-CUSTOMERS rule 20 does.

A destination moved that you were not trying to move. Read your diffed table. A route-map with a match clause broader than intended, or a set local-preference applied to a whole session rather than to a matched subset, will move everything on that session.

Cleanup

Everything here is policy on isolated bridges, so cleanup means restoring a known state rather than restoring service.

Step 1. If you want to keep the topology for the BGP break/fix scenarios or the capstone, stop here and save the current configuration on each router with save. Note in your journal that the site is running with LOCAL_PREF 200 toward isp-b and the 65010:80 community toward isp-a, so the next person to look at it is not surprised.

Step 2. To return to the Lab 11 end state — filters in place, no traffic engineering — remove the policy this lab added, on site1:

configure
delete policy route-map FROM-ISP-B rule 10 set local-preference
delete policy route-map FROM-ISP-A rule 10 set local-preference
delete protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast route-map export
delete protocols bgp neighbor 192.0.2.6 address-family ipv4-unicast route-map export
delete policy route-map TO-ISP-A
delete policy route-map TO-ISP-B
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast prefix-list export ONLY-MINE
set protocols bgp neighbor 192.0.2.6 address-family ipv4-unicast prefix-list export ONLY-MINE
commit
save

and on isp-a:

configure
delete protocols bgp neighbor 192.0.2.1 address-family ipv4-unicast route-map import
delete policy route-map FROM-CUSTOMERS
delete policy community-list CUST-LOWPREF
commit
save

Note the order in the first block: the replacement export filter is set in the same commit that removes the route-map. Removing an export policy on its own, even for a few seconds, restores the unfiltered behaviour Lab 11 existed to stop — and on a production edge that is a route leak, not a gap.

Step 3. To reset the whole topology to its unfiltered baseline, load the file saved in Lab 11 on each router:

configure
load /config/pre-lab-11.boot
compare
commit
save

Step 4. To remove the topology entirely, follow Lab 11’s Cleanup Step 3.

What You Learned

• Outbound and inbound are different problems with different guarantees. LOCAL_PREF decided outbound outright, because the decision was yours. Inbound, you sent two different arguments to two different routers, and each one could have ignored you.
• LOCAL_PREF beats AS-path length, and that ordering is deliberate. You watched a longer path through an extra AS win, because the algorithm places your knowledge above its own proxy for cost.
• FRR installs one path and keeps the rest. Two paths in the BGP table, one route in the RIB, one nexthop in the kernel. The spare made the failover fast precisely because it had already been received and evaluated.
• Prepending is a weak argument and a community is a strong one. Prepending is compared at step four, so any provider with a customer-preference policy has already decided by the time your prepends are read. A published provider community works at step two — and only exists where the provider chose to offer it.
• Two inbound techniques at once can cancel out. The prepend toward isp-b removed the alternate path isp-a needed, because BGP does not advertise a path back toward the peer it came from. Inbound traffic engineering steers a chain of decisions, and the chain has feedback.
• The deliverable is the table and the registry, not the configuration. The configuration says what you set. The before/after table says what moved, and the community registry says what the values mean to somebody who was not in the room.