Configuration Restored onto Wrong Interface Mapping

Reported symptoms

Five tickets, five owners, ninety minutes, one router.

edge-02 failed on Tuesday afternoon — a hard power fault, the unit would not POST. The spare came out of the rack of pre-imaged replacements, the configuration was pulled from Git, loaded, committed and saved, and the data-centre technician moved the four cables socket for socket from the dead chassis to the new one. The restore took eleven minutes and the ticket was closed as successful.

Then:

• Transit. The eBGP session to ISP-A has been in Active ever since. ISP-A, chased, replies that their side is Idle and they have seen no TCP 179 from us at all. Their engineer suggests we check our firewall.
• Desktops. No client on the user VLAN gets a DHCP lease. The helpdesk queue fills with “no internet”. The desktop team files it against the DHCP service, which runs on this router, and asks whether the lease database was restored.
• Core. The OSPF adjacency to the core switch never forms. show ip ospf neighbor is empty. The network team’s theory is that the router-id or the area configuration did not survive the load.
• Management. In-band SSH to the management address times out. The only path onto the box is the serial console the technician left plugged in. This is filed against the out-of-band switch as a disabled port.
• DMZ. The DMZ monitoring host says its default gateway is unreachable. Its segment mirror is recording ARP who-has for 192.0.2.5 — a transit next-hop address that has no business existing on the DMZ — sourced from the router’s MAC.

And a sixth, which arrives from outside the organisation and is the most useful sentence in the whole incident: ISP-A appends a note saying their handoff port is up and is receiving ARP requests for an RFC 1918 address that does not belong on that circuit.

Underneath all of it, the thing that did not happen: nothing is down. Four interfaces up/up, four link lights, no error counters, no carrier transitions. The cabling was checked visually once and ruled out, which is exactly what a visual check is good for and exactly what it is not.

The estate

edge-02 is a VyOS 1.5 LTS router on a four-port 1U appliance. The faceplate sockets are labelled 1 to 4 and the cabling standard for this model is fixed across the estate:

The router is the DHCP server for the user VLAN and the default gateway for the DMZ. Local AS is 64512. The whole configuration lives in Git and is identical in structure to the other eleven routers of this model in the estate, which is the point of having a standard.

The appliance has two NICs: an onboard dual-port controller and a dual-port card in a riser slot. Which pair the kernel enumerates first depends on where the card is fitted, and the spares kit does not specify a riser.

Evidence provided

vyos@edge-02:~$ show interfaces ethernet

Codes: S - State, L - Link, u - Up, D - Down, A - AdminDown
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.0.2.6/30                      u/u  WAN-ISP-A
eth1             -                                 u/u  CORE-TRUNK
eth1.10          10.42.10.1/24                     u/u  USERS
eth1.20          10.42.20.1/24                     u/u  SERVERS
eth2             10.42.30.1/24                     u/u  DMZ
eth3             10.42.99.20/24                    u/u  OOB-MGMT

vyos@edge-02:~$ show ip bgp summary

Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
192.0.2.5       4  64500         0        0    never          Active

vyos@edge-02:~$ ip neigh show dev eth0

192.0.2.5 dev eth0  FAILED

vyos@edge-02:~$ monitor traffic interface eth0

ARP, Request who-has 10.42.30.1 tell 10.42.30.41, length 28
ARP, Request who-has 10.42.30.1 tell 10.42.30.17, length 28
IP 10.42.30.41.5353 > 224.0.0.251.5353: 0 PTR (QM)? _services._dns-sd._udp.local.
ARP, Request who-has 192.0.2.5 tell 192.0.2.6, length 28

vyos@edge-02:~$ ethtool -i eth0

driver: igb
version: 5.15.0
firmware-version: 3.25, 0x8000058e
bus-info: 0000:01:00.0

And the kernel view, which is where the two NICs give themselves away — the vendor prefixes group the ports in pairs, and the pairs are not where the inventory puts them:

vyos@edge-02:~$ ip link show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
    link/ether 90:e2:ba:41:0c:80 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
    link/ether 90:e2:ba:41:0c:81 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
    link/ether 3c:ec:ef:19:a3:44 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
    link/ether 3c:ec:ef:19:a3:45 brd ff:ff:ff:ff:ff:ff

The restore ticket contains one more line, added by the engineer who did the work and written in the tone of someone tidying up a small obstacle:

hw-id lines removed from config.boot, would not commit otherwise

Work the evidence before reading on

The difficulty is that the configuration is right. Read it line by line against the design and you will find no error, which is why three of the five teams re-read it and moved on.

1. Five services broke at the same instant and the instant was a hardware swap. Ask what all five have in common that is not the configuration file.
2. show ip bgp summary says Active, not Idle. Those mean different things. Active says this router is trying to open a TCP session and getting nowhere. Combine that with the FAILED neighbour entry and you have a layer-two statement, not a BGP one.
3. Read the monitor traffic output by its source addresses rather than by what you expected to see there. The interface addressed 192.0.2.6/30 is watching 10.42.30.0/24 talk to itself.
4. ISP-A is reporting RFC 1918 ARP on the handoff. Which of this router’s interfaces could possibly emit that, and what does that imply about which wire it is on?
5. Compare the two bus-info values against the inventory rows for the failed unit, then ask what in a VyOS configuration is supposed to make the interface names independent of bus position — and whether it is still present.

Before continuing: the configuration is correct and the cables were moved correctly. What is the third thing, and where was it written down?

Root cause

1. hw-id was the mapping, and it was deleted

A VyOS configuration describes roles. set interfaces ethernet eth0 address 192.0.2.6/30 says what eth0 is for; it says nothing about which piece of copper eth0 is. The one directive that closes that gap is hw-id, which pins a named interface to a specific NIC MAC address, and it is written into /config/config.boot for every Ethernet interface.

That is why the restore would not commit. The hw-id values in the file name four MAC addresses that exist only on a chassis now in a returns box. VyOS refused, correctly, and the refusal was the system telling the truth: this configuration is for different hardware.

Deleting the four lines silenced the message without answering it. With hw-id gone, nothing in the configuration constrains the mapping, and VyOS presents whatever the kernel enumerated as eth0, eth1, eth2, eth3, in bus order.

2. The two chassis do not enumerate the same way

Both units are the same model. Both have an onboard dual-port controller and a dual-port card in a riser. On the failed unit the card was in the second riser slot and enumerated after the onboard pair; on the replacement, built from the spares kit by someone else on a different day, it is in the first. The bus positions in the evidence say so plainly: eth0 is at 0000:01:00.0 where the inventory records 0000:02:00.0, and the MAC vendor prefixes confirm the pairs have swapped ends.

So the mapping on the new chassis is the mirror of the old one:

The technician did the right thing. Socket for socket is the correct instruction; it was correct on every previous replacement. It is the faceplate labels that stopped meaning what they meant.

3. Every symptom falls out of one line of that table

• BGP Active. eth0 holds 192.0.2.6/30 and sits on the DMZ. It ARPs for 192.0.2.5, which is not on that segment, gets no answer, cannot open TCP 179 and retries forever. ISP-A sees nothing because nothing is being sent to them.
• No DHCP, no OSPF. eth1 carries vif 10 and vif 20 and is plugged into the out-of-band switch, which is an untagged access port. Tagged frames leave and are dropped. No user reaches the server, no core switch sends a hello.
• No in-band SSH. eth3 holds the management address and is on the core trunk. The address is not on the segment it needs to be on, so the out-of-band network cannot reach it.
• DMZ gateway down, and strange ARP on both ends. eth2 holds 10.42.30.1/24 and is on the ISP handoff. The DMZ has lost its gateway, and ISP-A is watching a router ARP at them in RFC 1918 space — which is the same event seen from the other side.

Resolution

1. Stay on the serial console for the whole of this. Both remediations move which wire the management address is on, and an in-band session would be the first casualty. Here there is no in-band session to lose, which is the only convenient thing about the situation.
2. Build the truth table before touching anything. For each of eth0 to eth3 record the MAC from ip link show, the bus position from ethtool -i ethN, and the segment actually present on the wire from monitor traffic interface ethN, read by the source addresses of the ARP and broadcast traffic. Four rows, measured. Every remediation below is a guess without them.
3. Choose the remediation deliberately, because the two are not equivalent. Re-cabling is immediate, needs no reboot and leaves the configuration byte-identical to Git - but it makes this the one chassis in the estate whose socket labels disagree with the model standard, and it hands the next replacement the same confusion in reverse. Pinning with hw-id keeps the cabling standard true for everyone, and puts the chassis-specific data back where the design always had it.
4. For the durable fix, set hw-id from the measured table so that each name lands on its intended socket: set interfaces ethernet eth0 hw-id 3c:ec:ef:19:a3:44 for the NIC in socket 1, and the equivalent for eth1, eth2 and eth3. Read the diff with compare before committing.
5. Commit and save, then reboot from the console. The interface rename is applied by udev at boot; do not assume the commit alone has moved the names. With this router carrying no traffic at all, the reboot costs nothing, which is what makes the durable fix the cheap one tonight.
6. Re-run the truth table after the reboot. This is the step that proves the fix rather than hoping for it, and it is the one that gets skipped because by this point the services have started coming back and the room relaxes.
7. If the window closes before the mapping is proven, hold: keep the router out of the traffic path with a named owner and an end time on the hold. A half-mapped router that forwards some traffic some of the time is worse than a router that is honestly down, because it turns a clean outage into an intermittent one.

The configuration change itself is four lines and reads as unremarkable, which is the point — the work was the measurement, not the typing:

set interfaces ethernet eth0 hw-id 3c:ec:ef:19:a3:44
set interfaces ethernet eth1 hw-id 3c:ec:ef:19:a3:45
set interfaces ethernet eth2 hw-id 90:e2:ba:41:0c:80
set interfaces ethernet eth3 hw-id 90:e2:ba:41:0c:81

Verification

1. The truth table is right. For every interface, monitor traffic interface ethN shows source addresses belonging to the subnet that interface is configured for. This is the check that would have prevented the incident, and it is the one that closes it.
2. Transit is on the transit wire. ip neigh show dev eth0 shows 192.0.2.5 REACHABLE rather than FAILED, and show ip bgp summary shows the session Established with a non-zero prefix count.
3. The core adjacency is up. show ip ospf neighbor shows Full with the core switch on the expected VLAN interface.
4. A real client on the user VLAN holds a real lease, confirmed from the server side rather than from a user reporting that it works now.
5. In-band SSH from the out-of-band network succeeds. Do not release the serial console until it does - the console is the only thing standing between you and a second trip to the data centre.
6. ISP-A confirms they no longer see RFC 1918 ARP on the handoff port. This is a stronger statement than "BGP came up", because it says the transit interface is on the transit wire rather than merely that a session found a path.
7. The mapping survives a boot. Reboot once more while you still have the console and re-run the truth table. A mapping that is correct now and wrong after the next unplanned reboot is not a fix.
8. All five original tickets close on their own evidence. One cause does not entitle you to close five tickets on one check.

Prevention

• The mapping is chassis data, not configuration data. The configuration describes roles; the inventory has to describe wires. One row per port per unit — socket label, MAC, bus position, interface name, role — and it is per unit, not per model, because this incident is precisely two units of the same model disagreeing.
• Re-derive hw-id, never delete it. Make it an explicit, named step in the different-appliance restore procedure, with the new chassis’s MACs read off the running system. The restore procedure should be unable to reach commit without it.
• Put the layer-two identity check before the commit. Reading the source addresses on each interface takes four commands and catches this entire class of defect while the router is still out of the path. As a post-change check it catches the same defect after five teams have been paged.
• Alert on the router, not only on the services. “All interfaces up and zero routing adjacencies” is a state worth naming. Every alert that fired here described a service someone else owned, which is how five people spent ninety minutes each looking at a different quarter of one fault.
• Run the DR drill onto different hardware. A restore rehearsed onto the same appliance exercises the file, the repository and the procedure, and cannot surface this failure mode at all, because the failure mode only exists when the hardware changes.
• Label by role at the chassis. The faceplate says 1, 2, 3, 4. The cable should say TRANSIT, CORE, DMZ, OOB. A label that names the role travels with the cable; a label that names a position travels with the metal.