Lab: BGP Peering

Objective

By the end of this lab you will have built a two-router, two-AS topology from nothing, brought up a single eBGP session, and proved that one prefix crossed it — where “proved” means four different commands answering four different questions, not one summary line answering all of them badly. You will then break the session on purpose, twice, and identify each break from evidence the router produced rather than from your memory of what you typed.

The lab exists because “BGP is up” is the least informative sentence in network operations. It can mean the TCP socket exists. It can mean the session reached Established. It can mean the peer sent you routes. It can mean the route you care about is in the forwarding table. Those are four different states, they fail independently, and an operator who cannot tell them apart will spend an outage looking at the wrong layer.

Architecture

Two VyOS routers in two private ASNs (RFC 6996 reserves 64512–65534 for exactly this), joined by one point-to-point transit link. Each router carries a dummy interface holding the address space its AS owns, so neither one needs a downstream host to have something worth advertising.

   AS 65001                                    AS 65010
   host: site1                                 host: isp-a
   router-id 192.0.2.101                       router-id 192.0.2.102

   dum0 198.51.100.1/24                        dum0 203.0.113.1/24
        |                                           |
   +----+----------------+               +----------+---------+
   |     site1           |               |      isp-a         |
   |  eth1 192.0.2.1/30  +---------------+ eth1 192.0.2.2/30  |
   +---------------------+   vmbr90      +--------------------+
        |                  (transit,          |
   eth0 (management)        no uplink)   eth0 (management)
        |                                     |
   +----+-------------------------------------+----+
   |        vmbr0 — your lab management LAN        |
   +-----------------------------------------------+

Two networks, two jobs. vmbr0 carries management only: it is how you reach the routers and it is never touched by any step in this lab. vmbr90 is an isolated bridge with no physical uplink, and it carries the transit link and nothing else. Keeping them separate is what makes this lab safe to run on a shared hypervisor — the routing you are about to break has no path to anything real.

The addressing comes from the RFC 5737 documentation ranges throughout. 192.0.2.0/24 is the infrastructure range (link /30s and router IDs), 198.51.100.0/24 is the space AS 65001 owns, and 203.0.113.0/24 is the space AS 65010 owns. Labs 11 and 12 extend this same plan with a third router, so the addresses are worth writing into your journal now.

Requirements

• A hypervisor that can run two VMs with two NICs each. The commands below are written for Proxmox VE; any KVM/libvirt host works if you can give each VM one NIC on your management LAN and one on an isolated bridge.
• 2 GB RAM and 8 GB disk per router, so 4 GB and 16 GB in total. VyOS runs comfortably in 2 GB for a lab of this size.
• The VyOS 1.5 LTS ISO uploaded to hypervisor storage. Everything in this lab is written against the VyOS 1.5 (circinus) configuration tree with FRR 10.x underneath. On 1.3 the BGP tree is spelled differently (set protocols bgp 65001 ... rather than set protocols bgp system-as 65001) and the command blocks here will be rejected.
• Console access to both VMs — the Proxmox web console or qm terminal. Tasks 6 and 7 deliberately break routing. They do not touch the management interface, so you should not lose your SSH session; keep the console available anyway, because “should not” is not a plan.
• Roughly 90 minutes. The VM build is about 30 of them.

Scenario

You are standing up a small site that has just been allocated 198.51.100.0/24 and a private ASN, 65001. One transit provider, AS 65010, has confirmed the session details: their side is 192.0.2.2, yours is 192.0.2.1, the link is a /30, and the session is plain eBGP with no authentication and no multihop.

Nothing is configured yet. The two VMs do not exist. You will build the starting state, not inherit it.

Tasks

Task 1: Build the starting state

First the isolated bridge. On the Proxmox host, add a Linux bridge with no ports — that is what makes it isolated. This is config text for /etc/network/interfaces, not a command:

auto vmbr90
iface vmbr90 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

Apply it with ifreload -a, or reboot the node if your host does not have ifupdown2. Then create the two VMs:

# Run on the Proxmox host.
# Substitute your own values before running. The ISO volume ID must
# match exactly what `pvesm list local` prints for your upload.
ISO=local:iso/vyos-1.5-lts-amd64.iso
STORE=local-lvm

for VMID in 210 211; do
  qm create "$VMID" \
    --memory 2048 --cores 2 --sockets 1 \
    --net0 "virtio,bridge=vmbr0,firewall=0" \
    --net1 "virtio,bridge=vmbr90,firewall=0" \
    --scsihw virtio-scsi-single \
    --scsi0 "$STORE:8" \
    --ide2 "$ISO,media=cdrom" \
    --boot order=ide2 \
    --ostype l26
done

qm set 210 --name site1
qm set 211 --name isp-a

qm start 210
qm start 211

firewall=0 on every NIC matters more than it looks. Proxmox’s per-interface firewall is implemented on the host; leaving it enabled on a routing VM means the host silently drops transit packets the guest expects to forward, and the resulting symptom looks exactly like a routing bug inside VyOS.

Open the console on each VM, log in with the live-boot credentials, run install image, accept the defaults, then set the boot order back to the disk and detach the ISO:

# Run on the Proxmox host, after `install image` completes on both VMs.
for VMID in 210 211; do
  qm set "$VMID" --boot order=scsi0
  qm set "$VMID" --delete ide2
done

Task 2: Baseline both routers, and save the state you will restore

Configure identity, management and the transit link. On site1:

configure
set system host-name site1
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description "management - not part of the lab"
set interfaces ethernet eth1 address 192.0.2.1/30
set interfaces ethernet eth1 description "transit to isp-a"
set interfaces dummy dum0 address 198.51.100.1/24
set interfaces dummy dum0 description "AS 65001 address space"
set service ssh
commit
save

On isp-a, the mirror image:

configure
set system host-name isp-a
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description "management - not part of the lab"
set interfaces ethernet eth1 address 192.0.2.2/30
set interfaces ethernet eth1 description "transit to site1"
set interfaces dummy dum0 address 203.0.113.1/24
set interfaces dummy dum0 description "AS 65010 address space"
set service ssh
commit
save

If your management LAN has no DHCP, give eth0 a static address from your own management range instead. Everything else is unchanged.

Now save a named copy of this configuration on both routers. Cleanup restores from it, and a lab that cannot get back to its starting state is a lab you can only run once:

save /config/pre-lab.boot

Prove the link before you go near BGP:

$ ping 192.0.2.2 count 3

PING 192.0.2.2 (192.0.2.2) 56(84) bytes of data.
64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=0.512 ms
64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=0.436 ms
64 bytes from 192.0.2.2: icmp_seq=3 ttl=64 time=0.448 ms

--- 192.0.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2043ms

This is the one moment in the lab where ping is the right tool. It answers “can these two interfaces exchange IP packets”, which is a precondition for BGP and is not a test of BGP. Once the session exists, ping will keep succeeding through every failure you are about to create, which is precisely why it stops being useful.

Task 3: Bring up the session, and watch the state machine

Configure site1 first, deliberately, so that for a minute or two only one side is configured. That asymmetry is the point of this task.

configure
set protocols bgp system-as 65001
set protocols bgp parameters router-id 192.0.2.101
set protocols bgp parameters log-neighbor-changes
set protocols bgp neighbor 192.0.2.2 remote-as 65010
set protocols bgp neighbor 192.0.2.2 description "isp-a transit, lab session"
commit-confirm 5

commit-confirm 5 starts a five-minute timer; if you do not type confirm, the router reverts to the previous configuration on its own. Nothing in this task can lock you out — management is on eth0 — but the habit is the one that matters on a real edge router, where the session you are configuring is often the session you are connected over. Type confirm once the commit is in.

Now read the summary immediately, before the other side is configured:

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.101, local AS number 65001 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.2       4      65010         0         0        0    0    0    never       Active        0 isp-a transit, lab session

Active is the state to sit with for a moment. It does not mean “actively working”. It means the local speaker attempted the TCP connection and it did not settle — here because nothing is listening on port 179 at the far end yet. In production the same state has a much longer list of causes: wrong peer address, a return-path ACL, an unstable source address, an authentication mismatch. The lesson on the session lifecycle enumerates them; what this task gives you is the memory of what Active looks like when the cause is benign, so that you do not read it as an emergency later.

Now configure isp-a:

configure
set protocols bgp system-as 65010
set protocols bgp parameters router-id 192.0.2.102
set protocols bgp parameters log-neighbor-changes
set protocols bgp neighbor 192.0.2.1 remote-as 65001
set protocols bgp neighbor 192.0.2.1 description "site1 customer, lab session"
commit-confirm 5

Confirm, then watch the transition from the site1 side. Within one ConnectRetry interval the state should move through to Established.

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.101, local AS number 65001 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.2       4      65010         6         6        0    0    0 00:00:41            0        0 isp-a transit, lab session

The column that used to say Active now carries a number — zero, because neither router is originating anything yet. In FRR’s summary the State/PfxRcd column is overloaded: while the session is down it holds the state name, and once the session is up it holds the count of prefixes received. A dashboard that greps this column for the word “Established” will never match, because the healthy case does not print it — and a dashboard that alerts on a prefix count of zero will fire on a perfectly healthy session with nothing to say.

Task 4: Prove Established from transport evidence

The summary is a claim made by bgpd about itself. Corroborate it from two layers that bgpd does not control. First the socket, from the underlying Debian system:

$ ss -tnp | grep ':179'

ESTAB      0      0            192.0.2.1:44831        192.0.2.2:179    users:(("bgpd",pid=1421,fd=23))

Two things are worth noticing. The socket is owned by bgpd, which confirms which process holds the session. And the local port is ephemeral while the remote port is 179, which tells you that site1 initiated this connection rather than accepting it — useful later, because a peer configured passive on both ends produces a session that neither side ever opens.

Now the negotiated contract, which lives only in the neighbour detail:

$ show bgp neighbor 192.0.2.2

BGP neighbor is 192.0.2.2, remote AS 65010, local AS 65001, external link
Description: isp-a transit, lab session
BGP version 4, remote router ID 192.0.2.102, local router ID 192.0.2.101
BGP state = Established, up for 00:02:17
Last read 00:00:17, Last write 00:00:17
Hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
  4 Byte AS: advertised and received
  Route refresh: advertised and received
  Address Family IPv4 Unicast: advertised and received
Local host: 192.0.2.1, Local port: 44831
Foreign host: 192.0.2.2, Foreign port: 179

Record Hold time is 180, keepalive interval is 60 in your journal. That is the negotiated contract, and it is the lower of the two offered values — so a router configured for 180 will happily run at 30 if the peer asked for 30, and the operator who only reads their own configuration will believe 180. Route refresh: advertised and received is the other line to keep: it is what will let you re-apply an inbound filter in Lab 11 without resetting the session.

Task 5: Originate one prefix and prove it three times

On site1, originate the space AS 65001 owns:

configure
set protocols bgp address-family ipv4-unicast network 198.51.100.0/24
commit-confirm 5

Confirm. Then, on site1, check that the local speaker considers the prefix eligible and is offering it to the peer:

$ show ip bgp neighbors 192.0.2.2 advertised-routes

BGP table version is 2, local router ID is 192.0.2.101
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
             i internal, r RIB-failure, S Stale, R Removed
Origin codes:  i - IGP, e - EGP, ? - incomplete

 Network          Next Hop            Metric LocPrf Weight Path
*> 198.51.100.0/24  0.0.0.0                  0         32768 i

Total number of prefixes 1

Now move to isp-a, and ask the three questions separately.

Question one — did bgpd receive and accept the path?

$ show bgp ipv4 unicast 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (1 available, best #1, table default)
Advertised to non peer-group peers:
65001
  192.0.2.1 from 192.0.2.1 (192.0.2.101)
    Origin IGP, valid, external, best (First path received)
    Last update: Tue Aug 18 11:04:22 2026

valid, external, best is the answer. best means this path won best-path selection inside bgpd — trivially here, because it is the only path.

Question two — did zebra select it into the RIB?

$ show ip route 198.51.100.0/24

Routing entry for 198.51.100.0/24
Known via "bgp", distance 20, metric 0, best
Last update 00:01:12 ago
* 192.0.2.1, via eth1, weight 1

The * is load-bearing. It marks the nexthop as selected and installed. distance 20 is the eBGP administrative distance — if a static route to the same prefix existed here with the default distance of 1, it would win and this line would be present but not starred.

Question three — did the kernel accept it?

$ ip route show table 254 198.51.100.0/24

198.51.100.0/24 nhid 21 via 192.0.2.1 dev eth1 proto bgp metric 20

Table 254 is the kernel’s main routing table. proto bgp records which daemon installed the route. A prefix that appears in the first two outputs and not the third is the classic “BGP says it is fine and traffic is black-holed” incident, and the only way to see it is to ask the kernel directly.

Finish the chain with a data-plane test from isp-a:

$ ping 198.51.100.1 count 3

PING 198.51.100.1 (198.51.100.1) 56(84) bytes of data.
64 bytes from 198.51.100.1: icmp_seq=1 ttl=64 time=0.641 ms
64 bytes from 198.51.100.1: icmp_seq=2 ttl=64 time=0.492 ms
64 bytes from 198.51.100.1: icmp_seq=3 ttl=64 time=0.507 ms

--- 198.51.100.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2038ms

Four commands, four different claims, all now proven. Keep all four captures — the failure tasks are about watching them diverge.

Task 6: Break it — remote-AS mismatch

On isp-a, change the configured remote AS to a value site1 does not use, as if the provider had transcribed the customer’s ASN wrongly:

configure
set protocols bgp neighbor 192.0.2.1 remote-as 65099
commit-confirm 5

Do not type confirm. Let the timer run — this is also a chance to watch commit-confirm do its job. Before it expires, capture the evidence.

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.101, local AS number 65001 vrf-id 0
BGP table version 2
RIB entries 1, using 96 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.2       4      65010        14        16        0    0    0 00:00:09       Idle          0 isp-a transit, lab session

The summary tells you the session is down and nothing else. The reason lives in two other places. First the neighbour detail, which records why the session last dropped:

$ show bgp neighbor 192.0.2.2

BGP neighbor is 192.0.2.2, remote AS 65010, local AS 65001, external link
BGP state = Idle
Last reset 00:00:31,   Notification received (OPEN Message Error/Bad Peer AS)

Then the protocol log, which records the transitions in order:

$ show log protocol bgp

bgpd[1421]: %NOTIFICATION: received from neighbor 192.0.2.2 2/2 (OPEN Message Error/Bad Peer AS) 0 bytes
bgpd[1421]: %ADJCHANGE: neighbor 192.0.2.2(isp-a) in vrf default Down BGP Notification received

Bad Peer AS is the whole diagnosis, and note where it came from: the router that had the correct configuration. site1 is unmodified, yet site1 is where the answer is legible, because the NOTIFICATION was sent to it. This is the reflex the task is building — when a session will not come up, read the neighbour detail on both ends before reading either configuration.

ping 192.0.2.2 still succeeds throughout. Confirm that, and write down what it proves: that the path is fine and the disagreement is at the protocol layer.

Now wait out the commit-confirm timer. When it expires, isp-a reverts to remote-as 65001 on its own and the session re-establishes. Verify on isp-a that the configuration really did revert:

show configuration commands | match "neighbor 192.0.2.1"

Task 7: Break it differently — advertise a prefix you cannot reach

This failure is quieter, and it is the more instructive of the two. On site1, remove the address that backs the originated prefix:

configure
delete interfaces dummy dum0 address 198.51.100.1/24
commit

The connected route for 198.51.100.0/24 is now gone from site1. Check what happened to the advertisement:

$ show bgp ipv4 unicast 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (1 available, best #1, table default)
65001
  192.0.2.1 from 192.0.2.1 (192.0.2.101)
    Origin IGP, valid, external, best (First path received)

The path is still there. isp-a still installs it, still forwards toward it, and show bgp summary still shows one prefix received. Prove the consequence from the data plane:

$ ping 198.51.100.1 count 3

PING 198.51.100.1 (198.51.100.1) 56(84) bytes of data.

--- 198.51.100.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2041ms

A network statement is a statement of intent. By default it does not require the prefix to exist in the local RIB, so it will keep originating a prefix the router cannot reach — and every downstream router will believe it. VyOS exposes the control that changes that contract. On site1:

configure
set protocols bgp parameters network-import-check
commit

Now re-check from isp-a:

$ show bgp ipv4 unicast 198.51.100.0/24

% Network not in table

The absence is the correct answer. network-import-check makes an unbacked network statement visible as missing rather than as a black hole — which is a strictly better failure, because a missing prefix generates a call and a black hole generates a mystery.

Restore the address and confirm the prefix returns:

configure
set interfaces dummy dum0 address 198.51.100.1/24
commit
save

Validation

Work through each item and confirm the stated result before you call the lab done.

• show bgp summary on both routers shows one peer with a numeric State/PfxRcd value, not a state name.
• ss -tnp | grep ':179' on site1 shows one ESTAB socket owned by bgpd, with remote port 179.
• show bgp neighbor on site1 reports BGP state = Established and a negotiated hold time you can quote, along with Route refresh: advertised and received.
• On isp-a, all three of show bgp ipv4 unicast 198.51.100.0/24, show ip route 198.51.100.0/24 and ip route show table 254 198.51.100.0/24 return the prefix, and the RIB entry carries * on its nexthop.
• ping 198.51.100.1 count 3 from isp-a returns three replies.
• Your journal holds the Bad Peer AS reset reason from Task 6, and records that ping 192.0.2.2 succeeded while the session was down.
• Your journal records that in Task 7 the prefix stayed in all three tables on isp-a while being unreachable, and disappeared only after network-import-check was enabled.
• Answer in writing: of the four commands in Task 5, which ones still returned a healthy-looking result during Task 7? The answer is the reason this lab asks four questions instead of one.

Expected Outcome

Two routers, two ASNs, one eBGP session in Established, one prefix originated by AS 65001 and installed in the kernel FIB of AS 65010 with a working data path. site1 has network-import-check enabled, so its network statement is now backed by a reachability requirement. Both routers still hold /config/pre-lab.boot, so the topology can be reset to its baseline without a rebuild.

The lab journal contains, at minimum: the four Task 5 captures, the two Task 6 captures with the reset reason, the Task 7 before-and-after pair, and the written answer to the closing question.

Labs 11 and 12 extend this topology with a second upstream. If you plan to continue, leave both VMs running and skip the VM-removal step in Cleanup.

Troubleshooting

The session sits in Active with both sides configured. Check the peer address on each router against the address actually configured on the far interface. show bgp neighbor names the foreign host it is trying; show interfaces on the other router names what exists. A /30 transposition — .1 and .2 swapped — produces exactly this and is invisible in either configuration read on its own.

The session flaps every few seconds. Look at show log protocol bgp for the reset reason. A hold-timer expiry with a healthy ping points at packet loss on the transit link or at a hypervisor firewall dropping port 179; recheck that every NIC was created with firewall=0.

ss -tnp shows no socket at all. bgpd is not running or the neighbour was never committed. show configuration commands | match bgp tells you which. A committed neighbour with no socket and no log entries usually means the BGP process failed to start — journalctl -u frr on the VyOS shell has the reason.

The prefix is in show bgp ipv4 unicast on isp-a but not in show ip route. Something else owns the destination with a lower administrative distance, or the next hop does not resolve. show ip route 192.0.2.1 answers the second case: if the next hop is not reachable, the BGP path is held inactive and never offered to zebra.

The prefix is in show ip route with a * but not in ip route show table 254. The kernel refused the route. This is rare in a lab this small; when it happens, journalctl -u frr records the netlink error.

commit fails with a validation error on the BGP tree. You are almost certainly on VyOS 1.3, where the ASN is part of the node path (set protocols bgp 65001 neighbor ...) rather than a system-as leaf. Check with show version; this lab requires 1.5.

Cleanup

The transit link and the routing are entirely inside an isolated bridge, and management was never touched, so cleanup is about leaving a reproducible state rather than about restoring service.

Step 1. On each router, restore the configuration you saved in Task 2 and confirm the difference is empty:

configure
load /config/pre-lab.boot
compare
commit
save

compare before commit is the check that matters: it shows exactly what loading the file will change. An empty comparison means you are already at the baseline.

Step 2. If you are continuing to Lab 11 or 12, stop here and leave the VMs running. Both labs reuse site1 and isp-a and add one more router.

Step 3. Otherwise, remove the VMs and then the bridge.

# Run on the Proxmox host, after confirming these IDs with `qm list`.
for VMID in 210 211; do
  qm stop "$VMID"
  qm destroy "$VMID" --purge
done

Then remove the vmbr90 stanza from /etc/network/interfaces and run ifreload -a. Confirm with ip link show vmbr90, which should report that the device does not exist.

What You Learned

• “BGP is up” is four claims, and they fail independently. You proved a socket, a negotiated session, a selected route and an installed route with four different commands reading three different processes. Task 7 made three of them look healthy while the destination was unreachable.
• The FSM state is a diagnosis, not a status. Active with nothing listening at the far end looks identical to Active caused by a return-path ACL. Knowing the benign shape is what lets you recognise the malignant one.
• The reset reason is on the router that is configured correctly. Bad Peer AS appeared on site1, which you never touched, because the NOTIFICATION was sent to it. Read both ends before reading either configuration.
• ping stops being a BGP test the moment BGP exists. It succeeded through both induced failures. Its only role in this lab was proving the transit link before the session was configured.
• A network statement is intent, not reachability. network-import-check converts an unbacked origin from a black hole into a visible absence, and that is a choice you make per prefix with the consequences written down.